Laravel Queues With Supervisor

In this blog, we are going to take a look into Laravel Queues.
Queues allow you to defer the processing of a time-consuming task, such as sending an e-mail, until a later time which drastically speeds up web requests to your application.

For example, the sign-up page of your application. When the user fills in all the details you want to send a thank-you mail but as we all know sending mail can take some time and you don’t want the user to wait that long. This is when Queues come into play.
Queues take metadata(email address & email content in this case) and then perform the requested job at a later point in time. So you will send the response immediately and push the mail sending job onto a queue for later processing.
Sending email is just one example, any time-consuming process can be pushed onto the queue to speed up the response time, like transferring files, converting images etc.

Here in the image the laravel web app will receive the web request, prepare the metadata for the queue and will push the job to queue and then will send out immediate response  not waiting for the job to finish.
So, now we have the job into the queue how will it be pulled out and processed. Laravel has a queue listener(Laravel Worker) that runs in the background which listen to the queue and pulls out the job and process it.
If we don’t have a queue listener the jobs will never get processed.

Ok, now let’s understand this bit by bit.

Pushing jobs to queue

The queue is a list of jobs, so where does this list reside. For this laravel ships with queue drivers where we can save these lists. These queue drivers include

• Sync
• Database
• Amazon SQS: aws/aws-sdk-php ~3.0
• Beanstalkd: pda/pheanstalk ~3.0
• IronMQ: iron-io/iron_mq ~2.0|~4.0
• Redis: predis/predis ~1.0

In this blog, we will use database queue driver. In order to use this driver, we must have a table to hold these jobs. For creating the migration for this table use :

php artisan queue:table

After migration is generated just run the migrate Artisan Command.

php artisan migrate

This will create “jobs” table which will hold all the jobs of the queue. Finally, edit the environment settings (.env) and change the QUEUE_DRIVER to database.

The default controller located in app/Http/Controllers/Controller.php uses a DispatchesJobs trait. This trait provides several methods allowing you to conveniently push jobs onto the queue, such as the dispatch method.

 

<?php

namespace App\Http\Controllers;

use App\User;
use Illuminate\Http\Request;
use App\Jobs\SendThankYouMail;
use App\Http\Controllers\Controller;

class UserController extends Controller
{
    /**
     * Register the user and send a thank-you mail.
     *
     * @param  Request  $request
     * @return Response
     */
    public function signUp(Request $request)
    {
        // Create the user and save details in DB

        // Send thank-you mail with login details
        $this->dispatch(new SendThankYouMail($user)); 

        // Return thank-you view
    } 
}

This example dispatches the job from Controller but if you want to dispatch the job from somewhere else in your application just include the DispatchesJobs trait on any of the classes in your application to gain access to its various dispatch methods.

Pulling jobs from the queue

To process the next item on the queue, we can run artisan’s queue:work command. This command will take out one job and process it. If the queue is empty issuing this command will do nothing. But this command will execute the job only once and stop and then you will have to issue this command again to process another job.

For this reason,  we use artisan’s queue:listen command. This command keeps looking at the queue and as soon as one job is pushed to the queue it issues the queue:work command. The problem with this is if the server goes down, queue listener will also stop. Also, if the memory limit is exceeded then the queue listener stops. We can specify memory limit with the --memory flag when starting the listener, default is 128Mb.
To overcome this we can use some process monitor system[Supervisor] that can restart the queue listener.

Implementing Supervisor

Supervisor is a process monitor for the Linux operating system, and will automatically restart your queue:listen or queue:work commands if they fail. To install this on ubuntu issue the following command
sudo apt-get install supervisor

Supervisor configuration files are saved inside the /etc/supervisor/conf.d directory. Let’s create a laravel-worker.conf and give it execute permissions:
chmod +x laravel-worker.conf.

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work --tries=3 --daemon
autostart=true
autorestart=true
numprocs=2
stdout_logfile=/path/to/application/storage/logs/workers.log

A header value of [program:foo]describes a program with the name of “foo”. The name is used within client applications that control the processes that are created as a result of this configuration. It is an error to create a program section that does not have a name. The name must not include a colon character or a bracket character. The value of the name is used as the value for the %(program_name)s string expression expansion within other values where specified.

Section Values

• process_name : A Python string expression that is used to compose the supervisor process name for this process.
• command : The command that will be run when this program is started.
• autostart : If true, this program will start automatically when supervisord is started.
• autorestart : Specifies if supervisord should automatically restart a process if it exits when it is in the RUNNING state.
• numprocs : Supervisor will start as many instances of this program as named by numprocs. Note that if numprocs > 1, the process_name expression must include %(process_num)s
• stdout_logfile : Put process stdout output in this file

Let’s discuss the switches used in the artisan’s command written in this program.

php /path/to/artisan queue:work --tries=3 --daemon

Tries : This flag defines the maximum number of times a job should be attempted
Daemon : Running queue:work with --daemon option forces the queue worker to continue processing jobs without ever rebooting the framework. This results in a significant reduction of CPU usage when compared to the queue:listen command. As daemon queue workers do not restart the framework before processing each job, you should be careful to free any heavy resources before the job finishes.

Once the configuration file has been created, we have to update the Supervisor configuration and start the processes using the following commands:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start laravel-worker:*

And we are done. Now there will always be 2 [numpprocs] laravel workers looking for jobs in the queue. As soon as there is a job pushed it will be picked by the worker and processed.

Just to be clear supervisor has nothing to do with laravel it just issues the command to invoke laravel workers.

Some Tips

Deploying With Daemon Queue Listeners

Since daemon queue workers are long-lived processes, they will not pick up changes in the code without being restarted. So, before deploying the application we should always restart the queue.

php artisan queue:restart

Failed jobs

When the queue fails to process a job it will retry to process the job till the limit mentioned in retries switch is reached. It is not tried further. Laravel has a solution for this also. We can create migration & table for failed_jobs and the jobs that are failed will be pushed to this table. The name of the table can be configured via the config/queue.php configuration file.

To create a migration for the failed_jobs table, you may use the queue:failed-table command:

php artisan queue:failed-table

To view all of your failed jobs that have been inserted into your failed_jobs database table, you may use the queue:failed Artisan command:

php artisan queue:failed

To retry all of your failed jobs, use queue:retry with all as the ID:

php artisan queue:retry all

 

To delete all of your failed jobs, you may use the queue:flush command:

php artisan queue:flush

queue:listen or queue:work –daemon

When you are in development mode you can use queue:listen because you don’t want to restart the queue again and again. But in production, it is a good practice to use queue:work –daemon so that it does not reboot the framework again and again.

Hope this helps. Happy Coding 🙂

References:
http://laravelcoding.com/blog/laravel-5-beauty-sending-mail-and-using-queues
Laravel Docs
Supervisord