Tests fail in bitbucket pipeline, not local

Hi all,

I am running a test suite that utilizes Pest and Livewire testing. The tests run and pass locally, but in my pipeline environment they fail. If you haven't used Bitbucket pipelines before, it is similar to other CI/CD environs in that you just define a yml file with a docker image to run things.

The tests that involve simple classes and models pass, like testing set and get logic and being able to query the testing database. Migrations run as well. The tests fail when they involve Livewire and getting routes. I am getting 404s and 500s when I expect 200s, not seeing Livewire components in responses as a result, etcetera.

This tells me there is something wrong with how I have configured something in the environment, but I really don't know what that could be. I have a pipeline env file committed and the only real difference between that and my regular env file is the DB info. I am copying that into a regular .env file in a pipeline step so the env is basically the same as local except the DB.

Since it is a test, I expected to not need to actually run the application, like with artisan serve. I have tried running it just in case but it did not make a difference.

There are no other env files or variables set up that should be affecting this pipeline.

I am not sure what specific code to put that would help solve this. If anyone has a question or possible solution, happy to share more. Any advice would be appreciated. Thanks!

LaryAI

2 years ago

Level 58

It sounds like you're encountering a common issue where tests pass locally but fail in a CI/CD environment like Bitbucket Pipelines. This is often due to differences in the environment configuration. Here are a few steps you can take to troubleshoot and resolve the issue:

Check Environment Variables: Ensure that all necessary environment variables are set in the Bitbucket Pipeline environment. Differences in .env files or missing variables can lead to different behaviors.
Check Application Key: If you're using Laravel, make sure that you have set the APP_KEY in your pipeline environment. A missing or incorrect application key can cause unexpected behavior.
Database Configuration: Verify that the database configuration in the pipeline is correct and that the database can be accessed. Ensure migrations are running as expected.
File Permissions: Sometimes, file permissions can cause issues in CI/CD environments. Make sure that the web server user has the correct permissions to access the necessary files and directories.
Debugging: Add debugging information to your pipeline to get more insight into the failures. You can output logs or even the content of the response to see what's happening during the test.
Dependencies: Ensure that all dependencies are correctly installed in the pipeline environment. Sometimes, a missing or incorrect version of a package can cause tests to fail.
Artisan Commands: Although you mentioned that running artisan serve did not help, make sure to run php artisan config:clear and php artisan cache:clear to clear any cached configuration that might be causing issues.
Livewire Specific: Since the issue is with Livewire components, make sure that Livewire's assets are properly published and accessible in the pipeline environment.

Here's an example of a Bitbucket Pipeline configuration that you can use as a starting point:

image: php:7.4

pipelines:
  default:
    - step:
        caches:
          - composer
        script:
          - apt-get update && apt-get install -y unzip
          - docker-php-ext-install pdo_mysql
          - cp .env.pipeline .env
          - composer install --no-interaction --prefer-dist --optimize-autoloader
          - php artisan key:generate
          - php artisan migrate --env=testing
          - php artisan config:clear
          - php artisan cache:clear
          - ./vendor/bin/pest

Replace .env.pipeline with the name of your environment file for the pipeline. Adjust the PHP image version and extensions as necessary for your application.

If you continue to experience issues, please provide specific error messages or logs from the pipeline, as they can offer more clues to the underlying problem.

jaspercreel

2 years ago

Level 8

@LaryAI Here is an example of a test what would pass locally with a 200 status but fail in the pipeline with a 404

FAILED Tests\Feature\QuoteControllerTest > it can view a quote Expected response status code [200] but received 404. Failed asserting that 404 is identical to 200. at tests/Feature/QuoteControllerTest.php:25 21▕ $user = User::factory()->create(); 22▕ $quote = Quote::factory()->create(); 23▕ $this->actingAs($user); 24▕ $response = $this->get(route('quotes.edit', ['quote' => $quote])); ➜ 25▕ $response->assertStatus(200); 26▕ }); 27▕ 28▕ 29▕ it('can delete a quote', function () {

martinbean

2 years ago

Level 80

@jaspercreel We can’t really help without seeing any errors or code I’m afraid.

For tests that fail with 500 errors, add withoutExceptionHandling before re-running tests so that Laravel actually outputs the exception that caused to test to fail. That may give you pointers on where to look or what you may have forgotten to configure in an environment outside of your local development environment.

1 like

jaspercreel

2 years ago

Level 8

@martinbean Thanks, I did your suggestion and tried running withoutExceptionHandling. I saw I was getting a "NotFoundHTTPException" error. This is strange to me as I thought doing HTTP tests doesn't actually perform an HTTP request unless you are doing dusk. What I ended up doing was changing the APP_URL env to "http://localhost" from "127.0.0.1." This actually goes against the recommendations from bitbucket. However, once I did that and ran "npm run build" as part of the pipeline process, the tests passed.

Still confused as to why I needed to get the APP_URL right for testing and also why I needed to run npm run build, but I am glad it is working now.

martinbean

2 years ago

Level 80

@jaspercreel Are you using route groups that rely on a specific domain? As it sounds like you are, and that it is based on the app’s URL. As the request is hitting your app, but not matching a route if you’re getting an exception.

jaspercreel

2 years ago

Level 8

@martinbean Nope. This is what my web routes look like, the relevant ones at least. Just resource routes in auth middleware, and I am using $this->actingAs($user) in my tests:

Route::middleware(['auth'])->group(function () {
    Route::resource('shipments', ShipmentController::class);
    Route::resource('quotes', QuotesController::class);
    Route::resource('bookings', BookingController::class);
});

Please or to participate in this conversation.