Is it possible to use Laravel Dusk browser outside of testing?

Hi @Indemnity83 , the more complicated reason is that the site I will be scraping has several places where it is much easier to click as a user than to scrape using a tool like guzzle. They use javascript, ajax, and the require some pages to be accessed through POST calls.

So my example is really just to get started using the Browser as a method to do scraping, outside of testing. I didn't want the more complicated use cases I hope to tackle to distract from the primary goal of using this new way of pulling up page data.

I really think Dusk could be a powerful scraper if I can figure out how to use it from a console command!

Just thinking out loud here:

Dusk is an artisan command. You can call artisan commands from anywhere in your application (including from other commands). You can also run individual Dusk tests or groups of tests.

What is to stop you from just creating a Dusk "test" that does what you want?

There are two problems I can foresee. One is that you should not allow the DuskServiceProvider in production, because it registers privileged routes for logging in as any user (including admin!). This is a huge security risk.

The other problem is that Dusk cannot run without a browser. On your production web server, you would not normally expect a browser to be installed -- indeed there's normally no GUI at all.

I have used a similar tool (Codeception) to scrape content. But I used it locally and then just copied the data across. I was using it to fetch data from thousands of payment records from Worldpay (my own data!), since Worldpay had no API.

Codeception solves your first problem, since it does not register privileged login routes. But it doesn't solve the second problem. You will need to run your "tests" from an environment with a web browser.

Sites that allow you to use their content should offer an API. I do hope you're not planning to steal content, as that would be ever so naughty. ;-)

This might be of some interest:

https://github.com/FriendsOfPHP/Goutte

thanks @MikeHopley for the thoughts. I have found another way as well, but I think running the artisan command is likely a cleaner approach. For what it's worth, the following that I have implemented also seems to work:

In my Command class:

public function handle()
    {
        $testcase = new DuskShell;
        $testcase->prepare();
        $testcase->testBasicExample();
    }
}

class DuskShell extends DuskTestCase {
    
    public function testBasicExample()
    {
        Browser::$baseUrl = 'http://etcetcetc';
    ...
    }
}

Using this I can use the Browser object to navigate websites as someone with a browser would.

And thanks for the thoughts on the security risks, I will keep it in mind. It would be nice if every page with legally-usable data behind POST calls and ajax loading had an API too, but unfortunately not yet...

Using this I can use the Browser object to navigate websites as someone with a browser would.

Interesting! I'm not too familiar with Dusk myself, so I don't understand what's going on here. What context does it run under? Is it opening up a web browser, or is it using some other JS-supporting environment?

https://packagist.org/packages/travoltron/dusk-secure

I've gone ahead and ripped out all the route exposing user stuff to make a more viable Dusk-based scraper for us all.

I've gone ahead and ripped out all the route exposing user stuff to make a more viable Dusk-based scraper for us all.

Nice! :)

hey @travoltron , whoa, nice! I missed this when posted, I will check it out.

update: Hi @travoltron I installed your package (https://packagist.org/packages/travoltron/dusk-secure , as you list above), and it worked great! I was able to install it on a production server and use it to scrape websites from my production app.

I had been doing it anyway (security be damned), so this is much much better long term.

I tagged your answer as the Best answer. I really think people will go nuts for using Dusk to scrape instead of just test their own apps. The potential uses are incredible.

Thank you so much!

Just as a followup for anyone else using this package: It doesn't mention in the docs, but every instance of a Laravel\Dusk\ class should be replaced with an equivalent instance of the new class, i.e.

File: tests/DuskTestCase.php

use Laravel\Dusk\TestCase as BaseTestCase; should now be: use Travoltron\DuskSecure\TestCase as BaseTestCase;

etc.

I also tried to make my version

https://github.com/unglud/dusker