🐪 Perl Minion: Asynchronous Job Processing

Date Created: 2025-03-29
By: 16BitMiker
[ BACK.. ]

In today’s web applications, responsiveness is everything. When users submit forms, upload images, or trigger long-running tasks, blocking the main application thread causes delays and frustration.

That’s where 🐪 Minion comes in — a robust, database-backed job queue system for Perl that makes it easy to offload slow tasks to background workers, keeping your app snappy and scalable.

📋 What is Minion?

Minion is an asynchronous job queue system built for Perl applications. It allows you to queue up tasks (jobs) for background execution, leaving your main application to handle other requests without delay.

✅ Key Features

• 📦 Database-backed — Jobs persist across restarts

• 💾 Backend support — SQLite, PostgreSQL, MySQL, and more

• 👥 Distributed processing — Run workers on multiple hosts

• ⏱ Delayed execution — Schedule tasks for later

• 🔁 Automatic retries — Handle failures with retry logic

• ⬆️ Priorities and dependencies — Fine-grained control over job execution

• 🧭 Monitoring tools — Track job progress and status

🚀 Getting Started with Minion

Let’s walk through a full example that highlights how Minion works from task definition to job execution and monitoring.

▶️ Full Script: script.pl

​x
#!/usr/bin/env perl
use strict;
use warnings;
use Minion;

# Create a new Minion instance using SQLite for storage.
# The SQLite DB file 'jobs.db' will persist all job data.
my $minion = Minion->new(SQLite => 'sqlite:jobs.db');

# Define a task named 'sleep_task'
# This task simulates work by sleeping for a specified number of seconds.
$minion->add_task(sleep_task => sub ($job, $seconds) {
    # $job: the job object that represents the current job
    # $seconds: parameter passed when the job is enqueued

    sleep $seconds;  # Simulate time-consuming work

    # Mark the job as finished and store a result message
    $job->finish("Slept $seconds seconds");
});

# Main logic for submitting and monitoring a job
{
    # Enqueue a job to run sleep_task with a parameter of 5 seconds
    my $job_id = $minion->enqueue(sleep_task => [5]);

    print "Started sleep job: $job_id\n";

    # Simulate doing other work while the background job runs
    my $counter = 0;
    while (1) {
        # Get the current status of the job by its ID
        my $status = $minion->job($job_id)->info->{state};

        # Exit the loop if the job has finished
        last if $status eq 'finished';

        # Simulate foreground work while waiting
        $counter++;
        print "Did some work (count: $counter) while sleep running...\n";

        sleep 1;  # Sleep a bit to avoid busy-waiting
    }

    print "Sleep job finished!\n";
}

# If we run the script with the 'worker' argument, launch a background worker
if (@ARGV && $ARGV[0] eq 'worker') {
    # Start a dedicated worker process to fetch and process jobs
    $minion->worker->run;
}

▶️ Running the Example

You’ll need two terminals:

🧑‍💻 Terminal 1: Start the worker

xxxxxxxxxx
$ perl script.pl worker

🧑‍💻 Terminal 2: Submit a job

xxxxxxxxxx
$ perl script.pl
Started sleep job: 1
Did some work (count: 1) while sleep running...
...
Sleep job finished!

🔍 Understanding the Components

1️⃣ Job Queue Setup

xxxxxxxxxx
my $minion = Minion->new(SQLite => 'sqlite:jobs.db');

• Initializes Minion with SQLite backend.

• The jobs.db file holds all job metadata.

• For production, consider PostgreSQL:

xxxxxxxxxx
my $minion = Minion->new(Pg => 'postgresql://user:pass@localhost/myapp');

2️⃣ Task Definition

xxxxxxxxxx
$minion->add_task(sleep_task => sub ($job, $seconds) {
    sleep $seconds;
    $job->finish("Slept $seconds seconds");
});

• Defines a task named sleep_task.

• The task receives:

  • $job: job object with methods like finish, fail, retry.

  • $seconds: argument passed when queueing the task.

• sleep simulates a time-consuming operation.

• finish marks the job as completed and stores a result.

3️⃣ Job Enqueuing

xxxxxxxxxx
my $job_id = $minion->enqueue(sleep_task => [5]);

• Enqueues a job for the sleep_task with argument 5.

• Returns a job ID to track the job later.

• Arguments must be passed as an arrayref.

4️⃣ Worker Process

xxxxxxxxxx
$minion->worker->run;

• Starts a worker loop that:

  • Polls the queue for new jobs.

  • Executes tasks.

  • Handles retries or failures.

• Should be run in a separate process or thread.

📚 Advanced Features and Examples

🎚 Job Priorities

xxxxxxxxxx
# High-priority job (processed sooner)
$minion->enqueue(urgent_task => [] => {priority => 50});

# Low-priority job (processed later)
$minion->enqueue(background_cleanup => [] => {priority => -20});

• Priorities range from -100 to 100.

• Higher values run earlier when multiple jobs are available.

⏳ Delayed Jobs

xxxxxxxxxx
# Schedule a job to run 1 hour from now
$minion->enqueue(send_reminder => [] => {delay => 3600});

• Useful for reminders, notifications, or data syncs.

🔗 Job Dependencies

xxxxxxxxxx
my $parent_id = $minion->enqueue(generate_report => []);

# These jobs will only run after generate_report completes
$minion->enqueue(email_report => [] => {parents => [$parent_id]});
$minion->enqueue(archive_report => [] => {parents => [$parent_id]});

• Enables workflows with sequential tasks.

🔁 Retries and Error Handling

xxxxxxxxxx
$minion->add_task(api_call => sub ($job) {
    my $result = eval { make_api_request() };

    if ($@) {
        # After 3 tries, mark it as failed
        return $job->fail("API call failed after 3 attempts: $@")
            if $job->retries >= 2;

        # Retry with exponential backoff
        return $job->retry({delay => (2 ** $job->retries) * 60});
    }

    $job->finish({success => 1, data => $result});
});

• eval catches runtime errors.

• Retry up to 3 times with increasing delay.

• Avoids flooding external services after transient failures.

⛔ Rate Limiting with Guards

xxxxxxxxxx
$minion->add_task(api_call => sub ($job) {
    # Allow max 5 concurrent jobs in 60-second window
    return $job->retry({delay => 30})
        unless my $guard = $minion->guard('api_limit', 60, {limit => 5});

    # Perform the API call here
    # The guard is automatically released after this block exits
});

• Useful for APIs with rate limits.

• Prevents overloading third-party services.

🌟 Real-World Use Cases

📧 Email Processing

xxxxxxxxxx
$minion->add_task(send_email => sub ($job, $recipient, $subject, $body) {
    my $mailer = Email::Sender::Simple->new();
    my $email = Email::Simple->create(
        header => [To => $recipient, From => 'no-reply@example.com', Subject => $subject],
        body => $body,
    );

    eval { $mailer->send($email) };
    if ($@) {
        return $job->retry({delay => 300}) if $job->retries < 3;
        return $job->fail("Failed to send email: $@");
    }

    $job->finish("Email sent to $recipient");
});

• Offloads email sending to avoid blocking web requests.

🖼️ Image Processing

xxxxxxxxxx
$minion->add_task(resize_image => sub ($job, $image_path, $dimensions) {
    my $img = Image::Magick->new();
    $img->Read($image_path);
    $img->Resize(geometry => $dimensions);

    my $output_path = $image_path;
    $output_path =~ s/(\.\w+)$/_resized$1/;
    $img->Write($output_path);

    $job->finish({original => $image_path, resized => $output_path});
});

• Handles heavy image tasks asynchronously.

📊 Report Generation

xxxxxxxxxx
$minion->add_task(generate_report => sub ($job, $user_id, $report_type) {
    $job->note(progress => 0);  # Track progress

    my $data = fetch_report_data($user_id, $report_type);
    $job->note(progress => 50);

    my $report_path = generate_pdf_report($data);
    $job->note(progress => 90);

    store_report_location($user_id, $report_path);
    $job->finish({report_path => $report_path});
});

• Background reporting keeps your app fast and scalable.

🔍 Monitoring and Management

🧪 Admin UI (Mojolicious)

xxxxxxxxxx
$app->plugin('Minion::Admin');

• Web UI for monitoring jobs:

  • View pending, finished, failed jobs

  • Retry or delete jobs

  • Inspect job data

🔧 CLI Tools

xxxxxxxxxx
# List jobs
$ perl -MMojolicious::Plugin::Minion::Admin::Command -e 'jobs'

# Show job details
$ perl -MMojolicious::Plugin::Minion::Admin::Command -e 'job 123'

📝 Best Practices

✅ Keep tasks idempotent — jobs can run more than once
✅ Use delays + retries for transient errors
✅ Monitor queue size and worker utilization
✅ Use named queues to separate concerns:

xxxxxxxxxx
$minion->enqueue(send_email => [] => {queue => 'email'});
$minion->worker->run({queues => ['email']});

✅ Pass only lightweight data — use database IDs instead of full objects

🔄 Conclusion

Minion is a powerful tool for building scalable, responsive Perl applications. By moving slow or blocking operations to background jobs, you dramatically improve user experience and system resilience.

From image processing to APIs, email, and reporting — Minion handles it all with clean syntax, solid architecture, and production-ready features.

📚 Read More

• Minion Documentation

• Minion::Guide

• Mojolicious::Plugin::Minion

• Minion on GitHub

• Practical Minion Examples

Happy queuing! 🐪