Automate filling forms external websites

Published 2 weeks ago by Erwin

Hello,

We are looking for a method to fill forms on external websites. Those websites don't have an api and filling these form is too much work doing it manually.

So someone suggested to do this using Dusk. But I need to run this from a controller because the client wants to start it when he wants.

Dusk is used for testing and this isn’t testing. Anyone how has some suggestions or information on people with the same case?

Kind regards,

Erwin

bashy
bashy
2 weeks ago (1,083,200 XP)

Dusk uses a browser to run commands. It isn't just for testing. It's basically a scraper.

Erwin
Erwin
2 weeks ago (8,730 XP)

Do you know any tutorials where the use Dusk not as testing but for scaping sites? I found some sample code:

namespace App\Http\Controllers\Backend;

use Illuminate\Http\Request; use App\Http\Controllers\Controller; use App\Contract;

use Laravel\Dusk\Browser;

use Symfony\Component\Process\Process; use Symfony\Component\Process\ProcessBuilder; use Facebook\WebDriver\Chrome\ChromeOptions; use Facebook\WebDriver\Remote\RemoteWebDriver; use Facebook\WebDriver\Remote\DesiredCapabilities;

class WaterController extends Controller {

public function index()
{
    // Start chromedriver
    $chromeProcess = $this->buildChromeProcess();
    $chromeProcess->start();

    // Create a RemoteWebDriver Instance
    $driver = retry(5, function () {
        return $this->driver();
    }, 50);

    $browser = new Browser($driver);

    $browser->visit('https://www.website.nl/service/signup/sigugupproces#?step=1')
        ->type('PostalCode', 'Test')
        ->type('HouseNumber', 1);
}

public function driverSuffix() {
    switch (PHP_OS) {
        case 'Darwin':
            return 'mac';
        case 'WINNT':
            return 'win.exe';
        default:
            return 'linux';
    }
}
public function buildChromeProcess() {
    $driver = realpath(__DIR__.'/bin/chromedriver-'.$this->driverSuffix());
    if (realpath($driver) === false) {
        throw new RuntimeException("Invalid path to Chromedriver [{$driver}].");
    }

    $env = ['DISPLAY' => ':0'];
    if (PHP_OS === 'Darwin' || PHP_OS === 'WINNT') {
        $env = [];
    }
    return (new ProcessBuilder())
        ->setPrefix(realpath($driver))
        ->getProcess()
        ->setEnv($env);
}

public function driver() {
    return RemoteWebDriver::create(
        'http://localhost:9515', DesiredCapabilities::chrome(), 5000, 10000
    );
}

}

I run into the following problem:

Failed to connect to localhost port 9515: Connection refused

1/1) WebDriverCurlException

Curl error thrown for http POST to /session with params: {"desiredCapabilities":{"browserName":"chrome","platform":"ANY"}}

Failed to connect to localhost port 9515: Connection refused

I am working in an wamp environment on windows. Normal dusk testing works fine, but inside the controller I can't get it to work.

bashy
bashy
2 weeks ago (1,083,200 XP)

Yeah it's probably not the best idea to use it on the web. You also shouldn't use Dusk in production.

If you want to grab data from a URL, just curl the page and manipulate the contents that uses something like this https://github.com/paquettg/php-html-parser

Erwin
Erwin
2 weeks ago (8,730 XP)

Hmm oke.

But any other solutions to makes this manual job automated? It's a recurring job and the have to copy and past the data from the system to the supplier page to singup a person. So I dont want to scape anything I want to submit real data on the website.

bashy
bashy
2 weeks ago (1,083,200 XP)

I don't really know what you're asking now.

Erwin
Erwin
2 weeks ago (8,730 XP)

I will try to explain it.

We receive water contracts for houses. These contracts have to be registered at the water company that provides the water to the houses. This is an manual action for the company that we word for. So every day they receive around 30 new water contracts and they have to register them manually.

So normally we would think let's use there API so we can automate this process. But the problem is, these companies don't have API's.

That’s the reason we are looking for a solution to automate this. Dusk does this very well, except like you mentioned it shouldn’t be used on production.

What other options do we have?

topvillas

Guzzle.

bashy
bashy
2 weeks ago (1,083,200 XP)

@Erwin Oh, you want to populate form inputs on another website. Right.

  1. Try something like this: https://github.com/duncan3dc/dusk
  2. Use Laravel Dusk and just have a local build that you can actually run commands from a database. Can use groups as well php artisan dusk --group=filler:water
Erwin
Erwin
2 weeks ago (8,730 XP)

Thx @bashy. Will test it tomorrow.

Sign In or create a forum account to participate in this discussion.