warksit's avatar

Codeception tests fail when run together but pass individually

I have written a few accetance tests in codeception. My acceptance.suite.yml

class_name: AcceptanceTester
modules:
    enabled:
        - AcceptanceHelper
        - Laravel4
    config:
      Laravel4:
         filters: true

My tests all pass when run individually. But when I run them together the a test fails. It is failing because a Model Observer is not being fired on create. The observer is registered through a boot method of a trait of the model. I have 2 tests for different model and whichever test is run 2nd fails.

I am trying so hard to introduce testing into my development but have spent a day not coding trying to get these tests to pass and it is very frustrating! Any pointers would be very welcome.

0 likes
16 replies
mabasic's avatar

You don't need Laravel4 module for Acceptance testing, it is only for functional testing as mentioned in the module documentation.

You need PhpBrowser module and set your url. Filters are enabled by default when running acceptance testing because of PhpBrowser.

class_name: AcceptanceTester
modules:
    enabled:
        - PhpBrowser
        - AcceptanceHelper
    config:
        PhpBrowser:
            url: "http://your.app:8000"

See this for more info on Acceptance testing: http://codeception.com/docs/04-AcceptanceTests

Are you sure you need acceptance testing and not functional testing?

1 like
psmail's avatar

@mabasic I've had this kind of problem with functional tests too- and never did find a solution - so of its own that is probably not the fix.

warksit's avatar

@mabasic thanks for the pointers.

I now have the following suites set up:

class_name: AcceptanceTester
modules:
    enabled:
        - PHPBrowser
        - AcceptanceHelper
    config:
      PHPBrowser:
         url: 'http://app.admin:8000'
class_name: FunctionalTester
modules:
    enabled: [FunctionalHelper, Laravel4]
    Laravel4:
      filters: true

However despite the fact that I can login to my site manually I still can't get tests to pass. The functional test fails as my post login page has:

<meta http-equiv="REFRESH" content="900; URL={{route('pages.lock')}}">

and the test seems to be directed immediately to this page rather than stating on the logged in page.

The acceptance test fails with "Test code or tested code did not (only) close its own output buffers" which I think is caused by the login being redirected back to the login in page as invalid even though it isn't.

My test for both:

$I = new AcceptanceTester($scenario);
// or $I = new FunctionalTester($scenario);
$I->amOnPage('login');
$I->fillField('username','test');
$I->fillField('password','test');
$I->click('Sign me in');
$I->canSee('Dashboard');

@jeffreyway -> this might be why only 5% test! Spending this long trying to get such a basic test to work is enough to make me not want to write any! I see their value totally but this is excruciating.

mabasic's avatar

I know why it fails omg, let me just find the solution I posted earlier

Solution

http://laravel.io/forum/08-16-2014-risky-functional-tests-happening-from-amonpage-method-being-called-in-codeception?page=1#reply-14036

In short, run your test with -vvv to see more detailed error. Somewhere on the page you are testing you have left something unclosed or overclosed. In my case I was closing an empty section @section("test", "value")@stop

This is what I would do if I were you to find the cause:

  • disable login on app (allow all users to view the problematic page)
  • write a simple I->seeResponseCode(200) test and then remove parts of your code one by one until you pinpoint the part that is making problems
thepsion5's avatar

So, I've run into this issue, and it's a really annoying one. I'm not 100% sure on this, but I believe this is the cause.

Whenever Codeception (or PHPUnit) runs a test using the Laravel 4 module, it essentially boots up the framework before each test. However, all of Laravel's classes are only autoloaded once. This is normally perfectly fine, except in the case of Eloquent's boot methods. Why?

Take a look at the source code for an Eloquent model:

protected static $booted = array();

/* snip */

protected function bootIfNotBooted()
 {
  if ( ! isset(static::$booted[get_class($this)]))
  {
   static::$booted[get_class($this)] = true;

   $this->fireModelEvent('booting', false);

   static::boot();

   $this->fireModelEvent('booted', false);
  }
 }

The Eloquent model sets the $booted[classname] static property to true whenever it calls the boot method. But since the class is already loaded, the next time the framework is loaded, Eloquent still thinks the model is booted because the static property is already set.

This is why the tests run fine individually, but not when run collectively. When test #2 runs, Laravel still thinks Eloquent is booted, even though (since the framework still gets re-initialized every time) you still need to run the code in the boot method.

2 likes
psmail's avatar

@thepsion5 I reckon your the first person to really have addressed this type of question.

Can I ask a follow up though - how did you turn your diagnosis into solution? Apologies if you already highlighted it and that I might not know quite enough to have picked up on it. Cheers.

warksit's avatar

OK, slowly getting somewhere!

Functional tests were failing because I had the config wrong and therefore filters were not on. As I set a tenant id in the auth filter this was not being run. The error messages were not helpful but once I bought all lines back in to the cept file (had some stuff in FunctionalHelper) I got more useful messages and was able to debug. Corrected functional.suite.yml (I had missed the config: line):

class_name: FunctionalTester
modules:
    enabled: [FunctionalHelper, Laravel4]
    config:
        Laravel4:
            filters: true

@thepsion5 your explanation makes perfect sense. I therefore moved those tests into the acceptance suite and they now work (although run slower). Would you say this is a 'bug' in Laravel or Codeception or PHPUnit?! Or maybe just an incompatibility?

thepsion5's avatar
Level 25

@warksit - I wouldn't consider it a bug per se, it's kind of an unintended consequence of booting the framework up multiple times. Hopefully this implementation will be changed with Laravel 5.0, since without knowing the root cause it can be really hard to diagnose (I speak from experience).

@psmail - It's been awhile since I had to deal with it (I no longer use eloquent's boot static function for this very reason), but I think you can solve this one of two ways:

  1. Manually call your model's boot function in the setUp/_before function of your tests that interact with said model.
  2. Configure PHPUnit to run tests in separate processes.

I think either of those solutions should work, though I ended up going with #1.

psmail's avatar

@warksit (I have the very same tenant is issue) and @thepsion5 thanks to you both for putting your findings out there / responding. I still have some noodling to do, but I feel a lot further along now.

warksit's avatar

"Manually call your model's boot function in the setUp/_before function of your tests that interact with said model" has done the trick for me.

warksit's avatar

@mabasic I appreciate your input as you definitely helped me get my suites set up correctly and on the right path to resolution but felt that the boot issue that @thepsion5 pointed out was the key issue in my question. Really appreciate all the responses, thank you.

psmail's avatar

Forgive me - I'm a little behind on this one.

So the docs refer to 'parent::boot();' So are we saying that in the setup / _before function (that but I get) something like 'model::boot();', where 'model' is the name of the model in question?

Thanks for you patience.

warksit's avatar

exactly that (with namespace to model if relevant)

grandadevans's avatar

Hi Guys, Here's another spin on this in case somebody had the same issue as me. I am a functional suite consisting of 2 tests: they pass individually but fail when run as the suite. I tried all the suggestions above etc but my problem was different. I got

Can't be on page "/register":
Symfony\Component\HttpKernel\Exception\NotFoundHttpException:

This obviously (as you probably all know if you watch the videos) that the route cannot be found.

Codeceptoin Bug: This is a problem with Codeception

Solution: I found the answer on StackOverflow Basically you need to revert your codeception version back down to 2.0.2

"codeception/codeception": "2.0.2"

in your composer.json

That fixed it for me :-)

Please or to participate in this conversation.