fideloper

I think the solution I would try is:

1. Use multiple queue workers (to get concurrent jobs running).
   • This can be scaled out on multiple servers, with more queue workers running, as needed
2. Have each job take care a phone number. So each job will take one phone number and send each text message for that phone number (maybe with a call of sleep(1); between each SMS send)

Adding a 2nd second won't really help. Rather than being an issue of database performance (it might be, but likely isn't), it's mostly an issue of how databases work with transactions and timing.

Note: You'll very likely accidentally run a job more than once when you use multiple workers against the database driver for queue workers. I'd investigate using another queue worker type. Forge comes with Beanstalk on the server, that might be a good place to start.

I generally use AWS's SQS as it's very cheap and is one less piece of software to manage. However, it has it's own caveats that you may run into in terms of job run-time - you'll want to increase the default SQS queue's Visibility Timeout (or change it per-job to be something high).

The method I described above would have less queue jobs (one job per phone number) but take more time per job ( # SMSes to send * (1 second + api call time) ). Just be sure not to set a total job time allowed (don't set a --timeout flag on queue workers).

I use s3 for a .env file per environment. However I also generally host on AWS where I can apply permissions on the server itself (instance profiles) rather than generate a user with an access key/secret.

In any case, I download the .env file appropriate on each deploy (as part of the deploy script).

If you'd like you can encrypt them also (not just encrypting the s3 bucket but the actual file contents prior to uploading). Then your deploy script that copies the encrypted file would need to decrypt it (and know the encryption key).

I personally build a new server and move sites over, rather than do major updates.

Updates via release upgrades usually either "keep current custom configuration" or "overwrite current config to package maintainers" since it cannot merge them.

Creating a new server and moving sites over doesn't cause issues with old configuration being used, and lets you test everything ahead of time before switching DNS or moving a floating IP to the new server.

Try a curl request to the IP address of the server (you may need to do with from within a server in the same private network depending on your VPC setup).

Something like: curl -i http://172.68.12.111/, without any host headers, etc. That's essentially what the ELB will send along. The -i flag will show sent and returned HTTP headers, so you can see exactly what the response is.

My guess is that you'll see Nginx is returning a non-200 response if that curl request is going to a default Nginx site and not the one that's configured via a server_name.

Here (sorta random, I know) is a tweet that might clarify what I mean - how Nginx (and all web servers) use the Host header to decide which site to serve from any given HTTP request: https://twitter.com/fideloper/status/996731264337108992

You may want to look into APM monitoring on something like data dog or new relic. Application Monitoring should give you a better idea of things like requests over time to see if you're hitting issues when getting spikes in HTTP requests or what your other bottlenecks are (vs blackfire which may show a more limited view of just what's going on in code and not necessarily how that affects overall servesr).

There's a few resources in the PHP topic in servers for hackers that explains how you might want to decide what to set your PHP-FPM max_processes (etc) to. Specifically the top two video series linked there talk about it.

There's also my Scaling Laravel course which can give you pointers on how to scale out (altho what your bottlenecks are isn't specifically covered - generally you'll need some monitoring to inform you of that).

How large is your aws server? If it only has 1gb of ram, setting the memory limit of 2gb in php.ini won't help.

If you can't/don't want to also increase the server side to get more RAM available, you can enable SWAP as well.

See here for how to enable swap. It enables 4gb, although I typically only do 1GB myself.

I dont really plan on it right now, but I would learn kubernetes, at least enough to pay for a service to set it up for you and not feel lost :D

For anyone finding this 2 year old post, I wrote a solution to it here: https://serversforhackers.com/c/nginx-php-in-subdirectory

TL;DR:

server {
    listen 80 default_server;

    root /var/www/top/public;

    index index.html index.htm index.php;

    server_name _;

    location / {
        try_files $uri $uri/ /index.php$is_args$args;
    }

    location /nested {
        alias /var/www/nested/public;
        try_files $uri $uri/ @nested;

        location ~ \.php$ {
            include snippets/fastcgi-php.conf;
            fastcgi_param SCRIPT_FILENAME $request_filename;
            fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
        }
    }

    location @nested {
        rewrite /nested/(.*)$ /nested/index.php?/ last;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/var/run/php/php7.2-fpm.sock;
    }
}

Agree with @ejdelmonico

You can and should def not be worried about security when moving mysql to its own server.

Here's an article on mysql network security that will show you what to do to secure it.

Start by checking the laravel error log on the server(s) running the queue workers. You'll likely see errors popping up in the log file.

Supervisord may also have it's own logs related to starting/stopping the queue worker with some more information. Check /var/log/supervisor for log files, and check /etc/supervisor/conf.d for a supervisor configuration that starts your laravel queue worker, and see what files it's outputting log output to - you can then check that file for more info.

Key point: Always check the logs.

I'd start by moving the database to it's own server first. This is the biggest bang for your buck - you may even find you can downsize the application server after moving the database out (that's true for most apps, but it depends on your app).

After you move the database to a new server, use the Percona Config Wizard to get some better settings in relation to things like InnoDB Buffer Pool.

RDS is nice too, but it will cost a lot, and the speed of queries between a DO server and AWS will create some lag as well (Unless you move the application to AWS as well).

Here's some info on using RDS with MySQL if you want to see what it looks like.

Finally, I'm working on the Scaling Laravel course which will cover moving MySQL to its own server, optimizing it, and a lot more optimizations you can look at.

Oh, and if you want to see what you can do in your application to help reduce database load, check out the first module of the Scaling Laravel course here which covers database indexes and better use of Eloquent.

Hey everyone. I'd suggest a few things I've made on Docker:

• To get started: Docker in Development: https://serversforhackers.com/s/docker-in-development
• To see a better workflow with Docker in dev: https://serversforhackers.com/dockerized-app
• To see that workflow made into something official (and takes care of a lot of edge cases) check out https://vessel.shippingdocker.com/

Finally, to really dive deep on how Docker works (in a practical sense, I don't get into any linux-kernel-container stuff), check out https://serversforhackers.com/shipping-docker , which goes from basics, to development, to production with Docker.

We just attempted to use it, however it's still only compatible with MySQL 5.6 - We are using some json columns, so we ended up with MySQL on RDS.

Everything else seemed the same tho!

I did a live video setting up an Ubuntu server ( a digital ocean droplet, but Linode isn't any different if you get an ubuntu server running): https://www.youtube.com/watch?v=VQNrsMYCOFg

There will be a series up on serversforhackers.com with a guide on a setup for laravel pretty soon too.

Does "not working" mean you get errors? Or the jobs just don't seem to get worked on? (Or something else?)

They aren't meant to be called from a CRON task, altho calling it like with the withoutOverlapping() I guess could work. However I'm a bit suspect of that, I have a feeling calling it from the CRON task may be related.

Check this out to see about using supervisord (if your hosting supports it): https://serversforhackers.com/monitoring-processes-with-supervisord

Example supervisord config for a laravel queue worker: https://laravel.com/docs/5.4/queues#supervisor-configuration

It requires Java, which I haven't seen on any/many shared hosts. That being said, if you have sudo or root access, you'll be able to install what you need.

Sounds like you're on the right track. Here's some things you can try out as well.

Storage Dir

A common way to handle the storage directory (and this is how Envoyer does it I believe) is not to delete and re-create the storage directory, but instead to keep re-using a single one. This lets you persist logs and cache in one location without worrying about wiping them out.

You might have the storage dir in /var/www/storage. Then when you deploy, you can:

# Delete emtpy-ish storage dir from code repo
rm -rf /var/www/my-app/storage

# Symlink existing storage dir from real location into "my-app"
ln -sf /var/www/storage /var/www/my-app/storage

Bootstrap dir

The writable "stuff" in the bootstrap dir are all "optional," meaning any caches/files in there can actually safely be deleted and doesn't need to persist between deployments. Files in there will either be re-created after the first request into the app, or can be re-created by adding/re-running certain tasks on deploy, such as:

php artisan optimize
php artisan config:cache
php artisan route:cache

More info on those above three commands are available on this free mini-course here: https://serversforhackers.com/laravel-perf

I might be wrong about that: https://articles.braintreepayments.com/support/guides/payment-methods/paypal/overview

The latest versions of our client SDKs support PayPal for merchants in all countries where Braintree is available.

No problem! I'm sure you'll come to like Nginx - it's always seemed simpler in setup (although Apache and the stuff so often found in an htaccess file does seem more powerful on the surface).

Currently:

• Stripe does not: https://stripe.com/global
• Braintree also appears not to: https://www.braintreepayments.com/country-selection

That being said, Braintree will work with Paypal, which will likely accept payments from within Egypt. (More info here, potentially: https://www.paypal.com/eg/webapps/mpp/merchant)

It looks like the process is simpler now - the filenames are defined in the LogServiceProvider.

You can copy/paste that service provider file (or extend it) into a new file in your App namespace (say App\Providers\LogServiceProvider and replace that new service provider with the core LogServiceProvider referenced in config/app.php.

Then change the names in the few places it defines a file name within that new service provider, and you should be all set.

Note: You may have to add a use Illuminate\Log\Writer to the top of the new service provider since it references the Writer class in the core service provider that's already in the Illuminate\Log namespace.

Edit: Actually it is a little more complex, as LogServiceProvider is only registered within the Illuminate\Foundation\Application class. Luckily you can replace any class in Laravel. In this case, I would make a new class which extends the coreApplication class - perhaps in class App\Application. Then tweak the new Application class to use your own LogServiceProvider.

Once you make your own Application core class (App\Application), you can edit bootstrap/app.php to initialize an instance of your own Application class which contains the change to use your own LogServiceProvider.

(Literally nothing is impossible to change/over-ride in Laravel).

Have you used Apache with php-fpm, or always used Apache's modphp?

If you want to use Apache + php-fpm, I believe you'll get most of the benefits, and some experience with potential oddities with FPM (altho, there really aren't too many - the only one that comes to mind is running a Laravel app in a sub-directory. That's weird with Nginx + FPM, I'm not sure about Apache + FPM).

I believe that setup also let's you use one of the newer Apache MPM's as well.

Otherwise, I can only say generic things about Nginx. It's supremely stable.

The weirdest most-common issue that I've seen is about running PHP apps in subdirectories: https://asked.io/host-laravel-in-a-sub-directory-with-nginx

I meant there may have been an error (syntax error usually) in the configuration. The command sudo nginx -t will show you if there is.

There isn't really a best Nginx configuration to help with this particular issue.

I'm using amazon lightsail 5$ plan thanks

That, unfortunately, may be the issue - it seems that you're out of available RAM.

That being said, you may also have a bad Nginx configuration - run sudo nginx -t or check the /var/log/nginx/error.log to see more details on what it says.

Does your email provider have any logs for incoming connections? Perhaps the issue is on their end (or at the very least, you can see if their servers are receiving a connection from your app)

To clarify, CRON on aws isn't different than any other server. (It's something you can add to the server, but isn't offered as a AWS specific service, - altho you can approximate it with some of their offerings)

On most server types, you can:

1. Add an entry to /etc/crontab
2. Add a crontab-stye file to /etc/cron.d
3. Add an executable script to one of /etc/cron.daily, cron.hourly, cron.monthly, cron.weekly
4. Use crontab -e to edit a cron that's specific to your currently logged in user (or other users if you can use sudo, e.g. sudo crontab -u someuser -e

Then you can add an entry to run the scheduler, something like this (taken from the docs)

* * * * * php /path-to-your-project/artisan schedule:run >> /dev/null 2>&1

Yep, and you can disable services on newer servers using the systemctl command:

# Stop nginx
sudo service nginx stop # equivalent to sudo systemctl stop nginx`

# Disable nginx from starting back up on system boot
sudo systemctl disable nginx

(That might be nginx.service instead of just nginx, I forget)

Laravel queue workers (the PHP processing queue jobs) is just an instance of an application on the CLI (cd ~myapp.com && php artisan queue:listen ...).

So, each app can be configured to listen to a queue (preferably a separate queue per app). Then each app can add jobs to its own queue.

In that way, they jobs are segmented between apps - each app only puts job in and reads new job from their own queue.

This is usually just as simple as changing the name of the queue in the laravel queue config file. Forge's use of beanstalkd for the queue (which can handle having more than one named queue, or "pipe" in beanstalkd's language) and Supervisord for the laravel workers (queue:listen command) should work great for that.

So, in that setup, you won't have one queue worker listening for jobs from both applications - so you won't need any code deciding which domain was used. (I'm assuming they are two separate applications as you said!)

Laravel workers are basically instances of the application itself, running on the CLI instead of via PHP-FPM.

The nice part about this is that you can create just another of the one (or multiple) stateless app servers and run your workers on that! (Budget permitting, of course).

My strategy for that is:

1. Have a 1+ application servers behind a load balancer for web requests,
2. Have 1+ other application servers NOT in the load balancer rotation, used for queue workers.

To make it simpler, I'll sometimes provision exactly the same server for workers and web (so a worker server might also have nginx/php-fpm on it, just un-used).

Segfaults are pretty hard to figure out. Often it's related to memory usage.

Do you see any specific errors in /var/log/php7.1-fpm.log ? Does the server have any additional information in the syslog after that one message? (I'm not sure if papertrail can show you the log lines just below that one?)

Sounds to me like "de" is a typo somewhere, like a script running on when a user connects over SSH or something (something edited accidentally on that homestead box?)

You may need to remove a block of config that tries to redirect www to non-www that forge creates, but should work otherwise i think.

Did that work out for you?

On a practical side of how, I use much of the h5bp configuration for Nginx to set both gzip settings (see the nginx.conf file) and cache header settings (see h5bp/location/expires.conf).

It contains HSTS and other SSL configuration as well.

Now a days you can point multiple domains to a server with a single IP address (and use HTTPS) thanks to wide support for SNI.

The condition you have in your example (using a www version of a domain) will likely work.

In terms of Nginx, there's no issue there (again, thanks to wide support for SNI) - if the SSL cert used works for any domain used in the server_name section, it should be fine.

Note however, that I'm not entirely sure thats LetsEncrypt with Forge will setup the www and non-www version of the domain.

Forge usually sets up the www-version to redirect to the non-www version of a domain, so you may be fighting that.

Easiest way to find out is to test it out!

Nginx has it's own set of logs that will show Nginx errors, or HTTP access logs. (Often found in /var/log/nginx). These are configured within the Nginx server configuration, often per site.

PHP-FPM has it's own set of logs, which you are asking about. These related specifically to FPM, not PHP in general. These are error logs for FPM, which controls how PHP is run when it receives requests from Nginx. As you can see, that log defaults to /var/log/php-fpm.log.

Finally, Laravel has it's own logs, whose purpose are to record errors|info|debug and their stack traces, if any are given. These are the most interesting/important logs, and the ones you configure within Laravel/Lumen. These usually exist in path/to/laravel/storage/logs/laragel.log.

Are you aware of how the t2 series use CPU credits?

You may be eating into CPU credits during regular usage (very common on t2.small if your database is on the same server with moderate traffic).

Once your CPU goes above 20% usage, CPU credits will begin being used. (Info on that here).

Once you eat up your credits, the server may struggle to respond to requests under load as the CPU usage is then capped at that 20% (for t2.small).

We can see that in your graph where CPU usage gets pinned at 20% and doesn't go above it. In fact, the server probably needs to use more CPU but is then capped at that, and thus doesn't go above it.

(In fact, seeing that graph makes me think that the server is certainly too small - you should resize it higher to handle that traffic).

You can find graphs of CPU Credit usage and credits remaining within Cloudwatch.

One thing I do is use RDS, on a small size to reduce cost (RDS is a relatively expensive hosted database). Removing the database from the web server reduces CPU usage greatly - I can often use t2.nano's if I also remove redis from the server.

Another option of course is to use a larger server type in AWS. This is as simple as stopping the instance, changing the instance size, and restarting it. Use a larger t2 or consider going to a m4, where there is no CPU credit system.

@francois Apologies, I didn't mean to make it sound like you had that attitude!

IMO, Security is a pretty big boogey-man. We are smart enough to know that we don't know everything, which works against us when someone claims something is insecure, we don't really have a way to verify it for ourselves. Making this worse is the tendency for many to pile on the "THIS IS BAD AND EVERYTHING IS BURNING" bandwagon.

Nginx/PHP as user forge

This particular decision is definitely a trade-off between security and ease of use (as is every security-related decision you'll ever make).

Mis-information

There was some subtle(ish) mis-information stated in previous responses

1. Nginx/PHP-FPM does not run as user forge so that forge has permission to change nginx config files, add ssl certs, etc. Those files are, as normal, all owned as user root. Forge runs as user root when it does its maintenance tasks.
2. Nginx/PHP-FPM is run as user forge to make dev lives easier - it's all related to permissions.

forge user and permissions

A) The Forge server setup allows your application to write to the storage directory (cache, logs, etc). This is why you do not run into the white screen of death after deploying/creating an application in Forge.

B) It also lets you login and change files as needed (as user forge) without worrying about accidentally changing permissions on directories/files, as is so common when people run into permissions errors (e.g. using sudo... and accidentally creating/changing files to be owned as user root, or making files globally insecure).

The main drawback / concern to this decision (to my knowledge) is if you allow user uploads in your application.

If those files are in an accessible location to the public web, and the files aren't properly vetted, they could potentially contain scripts that can get run on the server by calling them via a web request - this is a common issue you see in Wordpress sites where plugins don't properly check files to ensure they don't contain anything the developer did not intend.

So, secure user input, especially file uploads - both in the admin, for authenticated users, and of course for public users.

And then worry less. Forge is still worth it.

Side note, I have no stake in whether you use Forge or not. However, I do run https://serversforhackers.com in the hope you you'll learn more about servers so you can worry less!

My guess (and what I see here in Texas, using Spectrum, formerly TWC) is that my IP address changes when the modem is rebooted (via losing power or when I unplug it).

Hi!

Do you know if it would be possible to see the redirect code used in PHP here as well? That could lead to a clue.

The Nginx config looks good.

In the meantime:

1. Within the server, run sudo service nginx configtest and/or sudo nginx -t to see if it complains about anything in the configuration
2. If the config checks out, manually reload nginx to make sure the new config is sucked in sudo service nginx reload
3. Try removing the location rewrite in the block listening on port 80, reloading nginx, and seeing if it works then
4. I'm assuming return 301 https://server_name$request_uri; is actually using your domain in reality instead of server_name being what's actually in the file - is that correct?

I don't think this is a reverse DNS issue, but possibly we just need more information about what you mean / what you need reverse DNS for (or why you're worried about it).

Can you elaborate more on that?

Hmm yeah, I think if a file doens't exist on the server, it may be getting sent to index.php with the usual nginx setup, so that could def make sense!

Yep.

Well, the user and group set on a file/dir is based on the user/group performing the action when adding/creating files.

The actual permissions they get (read, write, execute, and who can do those actions - user, group or other) is based on the umask settings.

Note that you can change umask settings, but only for log-in sessions that read your profile file (e.g. ~/.profile, ~/.bashrc, and similar). This does not include sFTP, nor SSH when instructing it to run a command instead of log in.

Default connect to redis will be defined in config/database.php.

It looks like the redis.conf file (probably /etc/redis/redis.conf) is where you can setup redis to listen on a TCP socket instead of just a Unix socket (if it's not already listening on both). Source: https://github.com/NodeRedis/node_redis/issues/204

Here's how I've done it:

https://serversforhackers.com/video/letsencrypt-for-free-easy-ssl-certificates

On some servers I've had to use the --force-renewal flag in the CRON script.

I think array($request->get('product_id')) might give you something like [[1,2,3]] instead of just [1,2,3]

Permissions is not everything - if your php.ini is not set to display_errors, then you can still get the blank white screen.

Related to permissions: Showing the permissions of the storage directory doesn't give us enough information to see if it's just a permissions issue because:

1. The storage/logs, and other subdirectories, may have other permissions set
2. We don't know what user PHP is being run as, which determines if it can write to the storage directory (and its subdirectories). If user www-data is trying to write to directories owned by user nicolas, and user nicolas is the only user that can write to the directories (as permissions drwxr-xr-x is set), then the app won't be able to write to that location