[L5] codecept broken since bootstrap/start.php removed

There will need to be a new Codeception module for Laravel 5.

Dang! I guess that may not happen for a while..

I guess I could test out Behat and see if that works..

I've been thinking about this.

What do we think is going to happen with testing tools with L5? How long did itvtake for the L4 module to come out following the release of L4? Is Behat currently working with L5? Is there another option for acceptance testing?

I've been wondering the same thing. I read somewhere on the forum that @jeffrey_way will start an series on testing soon (and probably this week). I hope he will cover testing in L5 in that series. I started using codeception aswell but at the moment that does not work so I am very interested in how to use PHPSpec, PHPUnit and Behat.

I understand that phpunit can be used for acceptance testing, leveraging selenium. It's verbose and certainly less human-readable. But I believe it is an option.

I just released a Laravel 5 module for Codeception, for more details see https://laracasts.com/discuss/channels/general-discussion/laravel-5-codeception-module.

@JanHenkG Thanks so much for building the new module! Hugely appreciated.

I installed the new module via composer, updated my functional yml and rebuilt codecept. I also made sure my testing DB was up to date.

On running however I get an error:

[ReflectionException]    
Class db does not exist  

I'm not too familiar with the internal workings of codecept so perhaps this is to do with the Db module not Laravel5.

I did a -vvv run and have pasted the output here:

http://pastebin.com/mfFdD5uX

If I manually test in the browser the functional test case passes.

Strangely it does appear, from the verbose output, that the scenario passes but then throws an error at the end. As a test I deleted my first scenario so that the second scenario would fire. It did fire and passed similar to the above pastebin, but as soon as it had passed the ReflectionException was thrown again.

Any idea what might be going on here?

@alexleonard I made some changes to the module, could you see if these fixed your problem? The current version is 1.0.3.

@JanHenkG You utter beauty!

updated and ran tests and they work! What a joyous relief that is to see my functional tests running again - my app was feeling very vulnerable!

@alexleonard it is good to hear the update fixed your problems.

It is such a huge relief to have codecept tests back in place again! I was beginning to feel a bit lost without them!

Have you got a flattr account or something - I'd love to buy you a coffee or a beer :)

@alexleonard I don't have flattr or something similar, but thx for the offer :)

@JanHenkG I wonder if something has changed regarding FormRequests in the framework.

After an L5 composer update this morning pulled in some laravel/framework changes, as far as I can see any functional tests that involve authentication are failing.

Sadly the _output is only showing Whoops, looks like something went wrong. and doesn't seem to provide any more details. However, any functional test I have which submits a form fails. But running a manual test in the browser the form submission succeeds.

I tried rebuilding with codecept build but no change.

I figured maybe it was an issue with the CSRF token, so added the following to one of my failing tests, just before form submission:

$token = $I->grabValueFrom("input[name='_token']");
dd(json_encode(['form-token' => $token, 'system-token' => \Session::all()]));

But the CLI output on running the test showed the tokens to be the same:

string(126) "{
    "form-token":"ppSjmKb56CVHZtoqQcS19V31cgC4406HAO6WCZkv",
    "system-token":{
           "_token":"ppSjmKb56CVHZtoqQcS19V31cgC4406HAO6WCZkv"}}"

So I don't think that's it either.

Any tests which don't submit forms run fine, and unit and api testers are also running fine.

Have you noticed this?

Is anyone else having issues with Codecept functional tests at the moment?

A little more poking and I'm noticing that it may be related to Session data not being stored correctly.

Any tests that submit a form just generate a generic Whoops, looks like something went wrong. message in the _output folder.

However, tests which don't submit a form but expect flash messages to be passed back to the view are also failing. Everything else in the test passes, but if I try to verify the flash message is displayed it fails.

My LogOutCept is quite simple

$I->amLoggedAs(['email' => 'test@example.com', 'password' => 'test']);
$I->click('Log out');
$I->seeCurrentUrlEquals('');
$I->see('You have been logged out');

This works up to the flash message check.

If I throw

dd(\Session::all());

just before the test for the logged out message, then I see my Session only has

array(1) {
    '_token' => string(40) "VkKomWBLCgGNubUuciurjKmz7pErUuphHrpninCp"
}

But if I dd session all on the view and manually test logging out, then that flash data exists:

[
    '_token' => 'kCo9c1GYtRl5SfQfBjoiqoAXXynHhWbAf6QryfwB',
    'url' => ['intended' => 'http://example.com/']
    'flash' => [
        'old' => ['notification']
        'new' => ['notification' => 'You have been logged out']
    ]
]

@alexleonard I haven't looked at the problem yet, but since you suspect the problem is session related it could be useful to take a look at my answer in the following thread:

https://laracasts.com/discuss/channels/general-discussion/latest-v5-laravelframework-csrfmiddleware-changes-broke-codeception-functional-tests

I will update the test project later today to see if the latest laravel changes maybe broke the tests.

@alexleonard The sample app also has problems when running the tests, a lot of TokenMismatchExceptions are thrown.

The CSRF check has changed. The token in the session is now checked by comparing it against the value of the X-XSRF-TOKEN request header. However, this value is always null in the tests. I assume this is a problem with my module, but I do not have the time to look into it today. I hope to look at it this weekend, but it could also be next week.

@JanHenkG That's cool. Thanks so much for having a look, really appreciate it. Obviously whenever you find the time is great!

Always feel very naked when I don't have tests (and to think a year ago I wasn't doing TDD at all!)

Thanks @JanHenkG for leading me onto the right path. I think I've found a solution. By letting the Laravel5 client implement TerminableInterface it can call terminate on the Illuminate kernel which will terminate the middleware and by extension, save the session.

I've made a pull request to your repo.

Here is the bits of code you can use @alexleonard to get it working.

use Symfony\Component\HttpKernel\TerminableInterface;

class Laravel5 extends Client implements HttpKernelInterface, TerminableInterface
{

    ...

    public function terminate(DomRequest $request, Response $response)
    {
        $this->httpKernel->terminate($request, $response);
    }

}

@jonsa Thanks for the pull request, saves me some work :) I tested your fix against the sample application, there were some issues but these were with the sample app itself, and after fixing those issues all tests ran as before.

@alexleonard I updated the module with the fix from jonsa, so you should be able to run your tests again.

Thanks so much guys. I was on another job the last while but delighted to come back to this one and have all my functional tests working again! Fantastic!

@JanHenkG I've just started a new L5 project and seem to be running into issues getting my session variables to be seen after form posts and so on by Codecept. I thought it would make sense to check back in from this thread as there's a bunch of Session related stuff in here.

So I have a very simple contact form I'm testing, and I want to make sure that when the form is submitted with no data, the user gets standard error messages back.

$I->submitForm('#pa-contact-form', []);
$I->amOnRoute('public.contact');
$I->see('Do not forget to include your name.');

Whilst it's working as expected when manually submitting the form in the browser (the session->has errors and displays them).

But if I dd(Session::all()) in codecept right after Codecept submits the form, there are no errors in the session.

So perhaps something has changed since the last time I set this up. Following your previous advice I set up a TestingServiceProvider and registered it in config/app.php after the SessionServiceProvider.

<?php namespace App\Providers;
use Illuminate\Support\ServiceProvider;
class TestingServiceProvider extends ServiceProvider {

         /**
            * Register the service provider.
            *
            * @return void
            */
        public function register()
        {
                if ($this->app->environment() == 'testing')
                {
                    $this->app['config']['session.driver'] = 'native';
                }
        }
}

If I dd(Config::get('session.driver') in Codecept it correctly returns 'native'. So I know the environment is being set to testing, and I know I'm using the native session driver, but I'm still not seeing the expected errors key in the session.

Perhaps I've forgotten some other step I needed to take to get it running.

I'm also using the L5 module for codecept functional tests (here's my functional.suite.yml)

class_name: FunctionalTester
modules:
    enabled: [Filesystem, Laravel5, Db]

Am I missing something obvious?

@alexleonard Form errors are removed from the session after reading them, this happens in a new middleware in Laravel5. So if you dump the session after displaying the form errors you will not see any.

For more details look at this issue: https://github.com/janhenkgerritsen/codeception-laravel5/issues/5

Hey @JanHenkG,

Thanks for the speedy response!

I checked out those new methods seeFormHasErrors and seeFormErrorMessages and tried them out, sadly both to no avail. I either get

Couldn't see form has errors :
Failed asserting that false is true.

or

Couldn't see form error messages {"email":"We need your email address so we can reply to you."}:
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-'We need your email address so we can reply to you.'
+''

I tried turning off my old TestingServiceProvider as well (which would set Session back to the current default which is just file, but no joy there (although perhaps this no longer makes a difference), I'm leaving the testing env on native for the time being anyway.

My packages are all up to date and I'm using dev-master for your L5 Codecept module, other than that it's pretty much a fresh L5 install.

I have a troubling feeling that I'm missing something patently obvious!

I haven't updated the sample app in a while. I plan to do that this week, maybe I run into some changes that will explain your problem. I will get back to you.

@alexleonard I updated the sample app to the latest development version of Laravel 5 and the tests still run. The form error tests can be found in the following file: https://github.com/janhenkgerritsen/codeception-laravel5-sample/blob/master/tests/functional/FormErrorsCept.php. Maybe you can find the problem by looking at what your code does different?

Also note that there are two different methods: seeFormErrorMessage($key, $errorMessage) and seeFormErrorMessages(array $bindings).

Hey @JanHenkG

I'm a total idiot! Can't believe I didn't spot this before but I after I submitted the form in the cept I had

$I->amOnRoute('public.contact');

Not

$I->seeCurrentRouteIs('public.contact');

Ahem... Sorry about that.. it's working now!

One other thing

Before I spotted that though I did notice something that seemed unusual. See the returns when I fire the following in my cept:

dd(\App::environment());
    // returns "local"
dd(\Config::get('session.driver'));
    // returns "array"

Pretty certain last time I did that I got "testing" and "native".

Not sure why it wouldn't be returning the 'testing' environment.

I tried editing the functional.suite.yml to include

class_name: FunctionalTester
modules:
    enabled: [Filesystem, Laravel5, Db]
config:
    Laravel5:
        environment: testing

But it still reports 'local'. Perhaps I need to do something new with .env files?

@alexleonard Laravel 5 now uses .env files for specifying environment specific configuration, you cannot override configuration files for a specific environment anymore. That is why I updated the module so that you can specify which .env file to load, for example a .env.testing file for testing specific configuration. I removed the functionality to set a specific environment for the application.

For some more details see my latest reply in this topic: https://laracasts.com/discuss/channels/general-discussion/laravel-5-codeception-module?page=2.

Thanks for getting back to me.

I tried that just now

Modified functional.suite.yml to be

class_name: FunctionalTester
modules:
    enabled: [Filesystem, Laravel5, Db]
config:
    Laravel5:
        environment_file: .env.testing

Created .env.testing in my project root:

APP_ENV=testing
APP_DEBUG=true
APP_KEY=some31charstring
DB_HOST=localhost
DB_DATABASE=testing
DB_USERNAME=root
DB_PASSWORD=
SESSION_DRIVER=native
MAIL_PRETEND=true

And then threw dd(\App::environment()); into my first cept which returned:

Functional Tests (12)
Trying to edit a page meta data (Admin/Meta/MetaEditCept)
"local"

Oh, I made sure to composer update (I'm using "janhenkgerritsen/codeception-laravel5": "dev-master"), and did a codecept build as well just in case.

Anything I might be missing?