Laravel Queues With Supervisor

In this blog, we are going to take a look into Laravel Queues.
Queues allow you to defer the processing of a time-consuming task, such as sending an e-mail, until a later time which drastically speeds up web requests to your application.

For example, the sign-up page of your application. When the user fills in all the details you want to send a thank-you mail but as we all know sending mail can take some time and you don’t want the user to wait that long. This is when Queues come into play.
Queues take metadata(email address & email content in this case) and then perform the requested job at a later point in time. So you will send the response immediately and push the mail sending job onto a queue for later processing.
Sending email is just one example, any time-consuming process can be pushed onto the queue to speed up the response time, like transferring files, converting images etc.

Screen Shot 2016-06-19 at 9.23.55 am

Here in the image the laravel web app will receive the web request, prepare the metadata for the queue and will push the job to queue and then will send out immediate response  not waiting for the job to finish.
So, now we have the job into the queue how will it be pulled out and processed. Laravel has a queue listener(Laravel Worker) that runs in the background which listen to the queue and pulls out the job and process it.
If we don’t have a queue listener the jobs will never get processed.

Ok, now let’s understand this bit by bit.

Pushing jobs to queue

The queue is a list of jobs, so where does this list reside. For this laravel ships with queue drivers where we can save these lists. These queue drivers include

  • Sync
  • Database
  • Amazon SQS: aws/aws-sdk-php ~3.0
  • Beanstalkd: pda/pheanstalk ~3.0
  • IronMQ: iron-io/iron_mq ~2.0|~4.0
  • Redis: predis/predis ~1.0

In this blog, we will use database queue driver. In order to use this driver, we must have a table to hold these jobs. For creating the migration for this table use :

php artisan queue:table

After migration is generated just run the migrate Artisan Command.

php artisan migrate

This will create “jobs” table which will hold all the jobs of the queue. Finally, edit the environment settings (.env) and change the QUEUE_DRIVER to database.

The default controller located in app/Http/Controllers/Controller.php uses a DispatchesJobs trait. This trait provides several methods allowing you to conveniently push jobs onto the queue, such as the dispatch method.

 

<?php

namespace App\Http\Controllers;

use App\User;
use Illuminate\Http\Request;
use App\Jobs\SendThankYouMail;
use App\Http\Controllers\Controller;

class UserController extends Controller
{
    /**
     * Register the user and send a thank-you mail.
     *
     * @param  Request  $request
     * @return Response
     */
    public function signUp(Request $request)
    {
        // Create the user and save details in DB

        // Send thank-you mail with login details
        $this->dispatch(new SendThankYouMail($user)); 

        // Return thank-you view
    } 
}

This example dispatches the job from Controller but if you want to dispatch the job from somewhere else in your application just include the DispatchesJobs trait on any of the classes in your application to gain access to its various dispatch methods.

Pulling jobs from the queue

To process the next item on the queue, we can run artisan’s queue:work command. This command will take out one job and process it. If the queue is empty issuing this command will do nothing. But this command will execute the job only once and stop and then you will have to issue this command again to process another job.

For this reason,  we use artisan’s queue:listen command. This command keeps looking at the queue and as soon as one job is pushed to the queue it issues the queue:work command. The problem with this is if the server goes down, queue listener will also stop. Also, if the memory limit is exceeded then the queue listener stops. We can specify memory limit with the --memory flag when starting the listener, default is 128Mb.
To overcome this we can use some process monitor system[Supervisor] that can restart the queue listener.

Implementing Supervisor

Supervisor is a process monitor for the Linux operating system, and will automatically restart your queue:listen or queue:work commands if they fail. To install this on ubuntu issue the following command
sudo apt-get install supervisor

Supervisor configuration files are saved inside the /etc/supervisor/conf.d directory. Let’s create a laravel-worker.conf and give it execute permissions:
chmod +x laravel-worker.conf.

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /path/to/artisan queue:work --tries=3 --daemon
autostart=true
autorestart=true
numprocs=2
stdout_logfile=/path/to/application/storage/logs/workers.log

A header value of [program:foo]describes a program with the name of “foo”. The name is used within client applications that control the processes that are created as a result of this configuration. It is an error to create a program section that does not have a name. The name must not include a colon character or a bracket character. The value of the name is used as the value for the %(program_name)s string expression expansion within other values where specified.

Section Values

  • process_name : A Python string expression that is used to compose the supervisor process name for this process.
  • command : The command that will be run when this program is started.
  • autostart : If true, this program will start automatically when supervisord is started.
  • autorestart : Specifies if supervisord should automatically restart a process if it exits when it is in the RUNNING state.
  • numprocs : Supervisor will start as many instances of this program as named by numprocs. Note that if numprocs > 1, the process_name expression must include %(process_num)s
  • stdout_logfile : Put process stdout output in this file

Let’s discuss the switches used in the artisan’s command written in this program.

php /path/to/artisan queue:work --tries=3 --daemon

Tries : This flag defines the maximum number of times a job should be attempted
Daemon : Running queue:work with --daemon option forces the queue worker to continue processing jobs without ever rebooting the framework. This results in a significant reduction of CPU usage when compared to the queue:listen command. As daemon queue workers do not restart the framework before processing each job, you should be careful to free any heavy resources before the job finishes.

Once the configuration file has been created, we have to update the Supervisor configuration and start the processes using the following commands:

sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start laravel-worker:*

And we are done. Now there will always be 2 [numpprocs] laravel workers looking for jobs in the queue. As soon as there is a job pushed it will be picked by the worker and processed.

Just to be clear supervisor has nothing to do with laravel it just issues the command to invoke laravel workers.

Some Tips

Deploying With Daemon Queue Listeners

Since daemon queue workers are long-lived processes, they will not pick up changes in the code without being restarted. So, before deploying the application we should always restart the queue.

php artisan queue:restart

Failed jobs

When the queue fails to process a job it will retry to process the job till the limit mentioned in retries switch is reached. It is not tried further. Laravel has a solution for this also. We can create migration & table for failed_jobs and the jobs that are failed will be pushed to this table. The name of the table can be configured via the config/queue.php configuration file.

To create a migration for the failed_jobs table, you may use the queue:failed-table command:

php artisan queue:failed-table

To view all of your failed jobs that have been inserted into your failed_jobs database table, you may use the queue:failed Artisan command:

php artisan queue:failed

To retry all of your failed jobs, use queue:retry with all as the ID:

php artisan queue:retry all

 

To delete all of your failed jobs, you may use the queue:flush command:

php artisan queue:flush

queue:listen or queue:work –daemon

When you are in development mode you can use queue:listen because you don’t want to restart the queue again and again. But in production, it is a good practice to use queue:work –daemon so that it does not reboot the framework again and again.

Hope this helps. Happy Coding 🙂

References:
http://laravelcoding.com/blog/laravel-5-beauty-sending-mail-and-using-queues
Laravel Docs
Supervisord

 

Advertisements

7 thoughts on “Laravel Queues With Supervisor

  1. i got lost from the point where you talked about installing a supervisor on ubuntu. so in my head i thought, supervisor is probably available on my mac because i use a mac book. My assumption is most likely wrong. put me right back on track on first, knowledge of supervisor on a mac book and the steps to creating a laravel worker and all that goodness that follows. i didn’t read to the end because i got stuck here. Thanks

    Liked by 1 person

    1. Hi There,

      Sorry for the late reply was caught up in some personal stuff. 🙂

      You can install supervisor on mac very easily. You can follow this link to install supervisor on mac.
      https://nicksergeant.com/running-supervisor-on-os-x/

      After installing supervisor all the steps will be same.

      FYI : Supervisor can only be installed on UNIX-like operating systems [Linux/Mac].

      Let me know if you face any issues in this.

      Thanks,
      Prabhat

      Like

    1. Hi Rahul,

      It is not possible to install supervisor on windows.
      http://supervisord.org/

      “Supervisor is a client/server system that allows its users to monitor and control a number of processes on UNIX-like operating systems.”

      You can check these links for alternatives, but I am not sure about this.
      So please read carefully and find a way to implement something similar to supervisor.

      https://stackoverflow.com/questions/7629813/is-there-windows-analog-to-supervisord

      https://www.quora.com/Is-there-an-alternative-to-Supervisord-on-Windows

      Thanks,
      Prabhat

      Like

  2. Hi,

    I’ve specified numprocs as 8 in the supervisor configuration. But everytime I dispatch a job, it basically processes the same job 8 times:

    [2017-08-14 02:00:11] Processed: App\Jobs\UpdateOrders
    [2017-08-14 03:00:09] Processed: App\Jobs\UpdateOrders
    [2017-08-14 04:00:07] Processed: App\Jobs\UpdateOrders
    [2017-08-14 05:00:06] Processed: App\Jobs\UpdateOrders
    [2017-08-14 06:00:10] Processed: App\Jobs\UpdateOrders
    [2017-08-14 07:00:08] Processed: App\Jobs\UpdateOrders
    [2017-08-14 08:00:09] Processed: App\Jobs\UpdateOrders
    [2017-08-14 09:06:39] Processed: App\Jobs\UpdateOrders

    Here’s what my supervisor config looks like:

    [program:product-manager]
    process_name=%(program_name)s_%(process_num)02d
    command=php artisan queue:listen
    numprocs=8
    autostart=true
    autorestart=true
    directory=/var/www/html
    redirect_stderr=true
    stdout_logfile=/var/www/logs/product-manager.log

    How do I set it so that it only uses one worker if I’ve only dispatched one job. Or utilize all 8 workers if I dispatched 8 jobs. Is that the way it works? Am I having a wrong idea about how multiple workers in laravel queue works? All I really want is to have each worker work on a separate job that I dispatched.

    Liked by 1 person

    1. If you have multiple workers running in laravel then it will take jobs from the queue one by one. Even if the tasks are same if they have been pushed to the queue it will be picked up by the next available worker.

      Looking at the logs the jobs are being pushed to the queue at a different time and being picked up by the workers. That means it is not processing the same job again, It is processing a different job of the same workflow.

      =====

      If you want to have workers working on a specific task I would recommend you to go through this link [https://laravel.com/docs/5.4/queues#customizing-the-queue-and-connection]

      “By pushing jobs to different queues, you may “categorize” your queued jobs and even prioritize how many workers you assign to various queues.”

      You can create different supervisor configuration to process different queues separately.
      To mention the queue name in supervisor you can use the flag “–queue=job-name” with “php artisan” command.

      Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s