README

Webscrape

Scrape web pages with a Laravel application.

Installation

You can install the package via composer:

composer require truercm/laravel-webscrape

You can publish and run the migrations with:

php artisan vendor:publish --tag="laravel-webscrape-migrations"
php artisan migrate

You can publish the config file with:

php artisan vendor:publish --tag="laravel-webscrape-config"

This is the contents of the published config file:

return [

    /*
    |--------------------------------------------------------------------------
    | Webscrape models
    |--------------------------------------------------------------------------
    */
    'models' => [

        /*
        |--------------------------------------------------------------------------
        | Subject model holds the credentials, target_id and the final scraping result
        |--------------------------------------------------------------------------
        */
        'subject' => TrueRcm\LaravelWebscrape\Models\CrawlSubject::class,

        /*
        |--------------------------------------------------------------------------
        | Target model stores the remote target, authentication url and processing job
        |--------------------------------------------------------------------------
        */
        'target' => TrueRcm\LaravelWebscrape\Models\CrawlTarget::class,

        /*
        |--------------------------------------------------------------------------
        | TargetUrl model collects all URLs for the Target
        |--------------------------------------------------------------------------
        */
        'target_url' => TrueRcm\LaravelWebscrape\Models\CrawlTargetUrl::class,

        /*
        |--------------------------------------------------------------------------
        | Url Result model stores processed results
        |--------------------------------------------------------------------------
        */
        'result' => TrueRcm\LaravelWebscrape\Models\CrawlResult::class,
    ],

    /*
     |--------------------------------------------------------------------------
     | Selenium driver url
     |--------------------------------------------------------------------------
     */
    'selenium_driver_url' => env('SELENIUM_DRIVER_URL', null),
];

Laravel Webcrawler uses Selenium to crawl the pages, so make sure you have it installed.

Usage

This is a generic package, you would need to implement all the crawling steps yourself.

The high concept overview involves:

1. Having a CrawlTarget - the model, containing the entry point to the list of pages you need to crawl
2. Crawl subject - a model that connects the credentials with the crawl target

Once you have registered a target, you can:

1. Initialize subject with credentials and target urls
2. Start remote url crawling and processing the result

    $crawlSubject = \TrueRcm\LaravelWebscrape\Actions\StoreCrawlSubject::run([
        'model_type' => App\Models\User::class,
        'model_id' => 1,
        'crawl_target_id' => 1,
        'credentials' => ['values' => 'that would be piped', 'into' => 'crawl target'],
]);

and from here:

resolve($crawlSubject->crawlTarget->crawling_job)
    ->dispatch($crawlSubject);

1. After the job is finished we have final result in CrawlSubject's result column

    $crawlSubject->result;

Testing

composer test

Changelog

Please see CHANGELOG for more information on what has changed recently.

Contributing

Please see CONTRIBUTING for details.

Security Vulnerabilities

Please review our security policy on how to report security vulnerabilities.

Credits

• Purnendu Chandan
• All Contributors

License

The MIT License (MIT). Please see License File for more information.