E2E Testing

Stoobly Scaffold E2E Testing - Questions & Answers

Stoobly scaffold provides first-class support for E2E testing frameworks like Playwright and Cypress, enabling you to record, mock, and validate API interactions during your end-to-end tests.

---

Plugin Support

Q: What E2E testing frameworks does Stoobly support?

A: Stoobly supports Playwright and Cypress through the --plugin option when creating an app.

Example:

# Create app with Playwright support
stoobly-agent scaffold app create my-app --plugin playwright

# Create app with Cypress support
stoobly-agent scaffold app create my-app --plugin cypress

# Create app with both plugins
stoobly-agent scaffold app create my-app --plugin playwright --plugin cypress

Q: What files are created when I use the --plugin option?

A: When you specify a plugin, Stoobly creates a custom Dockerfile and entrypoint script in the entrypoint/test/ directory specifically configured for that E2E framework.

Files created for Playwright:

• .stoobly/services/entrypoint/test/.Dockerfile.playwright - Stoobly-maintained base image that pulls in the stock Playwright image (override via PLAYWRIGHT_IMAGE)

• .stoobly/services/entrypoint/test/.entrypoint.sh - Entrypoint script with CA certificate setup

• .stoobly/services/entrypoint/test/docker-compose.yml - Docker compose configuration

• .stoobly/services/entrypoint/test/configure - Stoobly configuration script

• .stoobly/services/entrypoint/test/init - Initialization script

Files created for Cypress:

• .stoobly/services/entrypoint/test/.Dockerfile.cypress - Stoobly-maintained base image that pulls in the stock Cypress image (override via CYPRESS_IMAGE)

• .stoobly/services/entrypoint/test/docker-compose.yml - Docker compose configuration

• .stoobly/services/entrypoint/test/configure - Stoobly configuration script

• .stoobly/services/entrypoint/test/init - Initialization script

---

Initial Setup

Q: How do I create a scaffold script for E2E testing?

A: Create a shell script with only the scaffold creation commands. Include workflow commands as commented examples for reference.

Example:

#!/bin/bash

# Create scaffold app with Playwright plugin
stoobly-agent scaffold app create my-e2e-tests \
--run-on docker

# Add API service (API runs on localhost:4000, exposed as api.local:80)
# Note: --upstream-port is required when using --local and must differ from --port
# Run your API service on port 4000 (not 80)
stoobly-agent scaffold service create api \
--hostname api.local \
--port 80 \
--scheme http \
--local \
--upstream-port 4000

# Add UI service (remote, served on app.local:80 for testing)
stoobly-agent scaffold service create ui \
--hostname app.local \
--port 80 \
--scheme http \
--workflow test

# Example: Start record workflow
# stoobly-agent scaffold workflow up record \
# --context-dir-path ./my-e2e-tests \
# --hostname-install-confirm y \
# --ca-certs-install-confirm y

# Example: Start mock workflow
# stoobly-agent scaffold workflow up mock \
# --context-dir-path ./my-e2e-tests \
# --hostname-install-confirm y \
# --ca-certs-install-confirm y

# Example: Start test workflow
# stoobly-agent scaffold workflow up test \
# --context-dir-path ./my-e2e-tests \
# --hostname-install-confirm y \
# --ca-certs-install-confirm y

Q: How do I scaffold an app for Playwright E2E testing?

A: Create an app with the --plugin playwright option to set up Playwright-specific configurations.

Example:

# Create app with Playwright
stoobly-agent scaffold app create my-e2e-tests \
  --plugin playwright \
  --run-on docker

Q: How do I scaffold an app for Cypress E2E testing?

A: Create an app with the --plugin cypress option to set up Cypress-specific configurations.

Example:

# Create app with Cypress
stoobly-agent scaffold app create my-cypress-tests \
  --plugin cypress \
  --run-on docker

---

Frontend Service Setup

Q: How do I set up a frontend service to serve static assets for E2E tests?

A: Frontend services typically only need the test workflow since they serve static assets from fixtures rather than recording or mocking requests. Create the service with --workflow test and modify the test/init script to copy your built application to the public fixtures folder.

Example:

# Create frontend service with only test workflow
stoobly-agent scaffold service create frontend \
  --hostname app.local \
  --port 80 \
  --scheme http \
  --workflow test

Modify the init script to copy built assets:

# .stoobly/services/frontend/test/init
#!/bin/bash

# Copy built frontend assets to public fixtures folder
# This makes the application available for E2E tests
cp -r ./dist/. .stoobly/fixtures/public/

# The frontend service will serve these static files during E2E tests

Why only test workflow?

• Frontend services serve static assets, not dynamic API responses

• No need to record requests (no API logic to capture)

• No need to mock responses (serving files, not making requests)

• Test workflow is needed so E2E tests can request the application under test

---

Configuration Files

Q: What files should I modify to add my Playwright tests?

A: After scaffolding, you need to modify the entrypoint service docker-compose.yml to run your tests, the configure script to set up Stoobly rules, and your Playwright configuration to use the proxy. The example below shows how you would override the Stoobly-managed entrypoint.playwright service definition—only do this when you need additional customizations.

Example - Modify docker-compose.yml:

# .stoobly/services/entrypoint/test/docker-compose.yml
services:
  entrypoint.playwright:
    build:
      context: .
      dockerfile: .Dockerfile.playwright
      args:
        PLAYWRIGHT_IMAGE: mcr.microsoft.com/playwright:v1.40.0-jammy
        USER_ID: ${USER_ID}
    working_dir: /home/stoobly/tests
    volumes:
      - ../../..:/home/stoobly/tests    # Mount your test directory
    depends_on:
      entrypoint.configure:
        condition: service_completed_successfully
    networks:
      app.ingress: {}
    environment:
      - HTTP_PROXY=http://gateway:80
      - HTTPS_PROXY=http://gateway:80
      - NO_PROXY=localhost,127.0.0.1
    profiles:
      - ${WORKFLOW_NAME}
    command: npx playwright test          # Your test command

Example - Modify configure script:

# .stoobly/services/entrypoint/test/configure
#!/bin/bash

# Rewrite production URLs to local services
stoobly-agent config rewrite set \
  --pattern "https://api.production.com/.*" \
  --method GET --method POST \
  --mode test \
  --hostname api.local

# Exclude third-party tracking
stoobly-agent config firewall set \
  --pattern "https://.*analytics.com/.*" \
  --method GET \
  --mode test \
  --action exclude

Example - Create Playwright config:

// playwright.config.ts (in your project root)
import { defineConfig } from '@playwright/test';

export default defineConfig({
  use: {
    proxy: {
      server: 'http://gateway:80',
    },
    ignoreHTTPSErrors: true,
  },
});

Q: What files should I modify to add my Cypress tests?

A: For Cypress, modify the entrypoint service docker-compose.yml and Cypress configuration to work with Stoobly's proxy. As with Playwright, the example below overrides the managed entrypoint.cypress service definition—only override it when you need custom behavior.

Example - Modify docker-compose.yml:

# .stoobly/services/entrypoint/test/docker-compose.yml
services:
  entrypoint.cypress:
    build:
      context: .
      dockerfile: .Dockerfile.cypress
      args:
        CYPRESS_IMAGE: cypress/included:13.6.0
        USER_ID: ${USER_ID}
    working_dir: /home/stoobly/e2e
    volumes:
      - ../../..:/home/stoobly/e2e        # Mount your test directory
    depends_on:
      entrypoint.configure:
        condition: service_completed_successfully
    networks:
      app.ingress: {}
    environment:
      - HTTP_PROXY=http://gateway:80
      - HTTPS_PROXY=http://gateway:80
    profiles:
      - ${WORKFLOW_NAME}
    command: cypress run                   # Your test command

Example - Create Cypress config:

// cypress.config.js (in your project root)
const { defineConfig } = require('cypress');

module.exports = defineConfig({
  e2e: {
    env: {
      HTTP_PROXY: 'http://gateway:80',
      HTTPS_PROXY: 'http://gateway:80',
    },
  },
});

Q: How do I add custom npm packages for my tests?

A: Don’t edit the Stoobly-maintained Dockerfiles (files beginning with a dot). Instead, point PLAYWRIGHT_IMAGE or CYPRESS_IMAGE at a custom image that already includes your dependencies.

# Example: override Playwright image when running a workflow
PLAYWRIGHT_IMAGE=my-org/playwright-with-helpers:latest \
  make -f .stoobly/services/Makefile test

# For Cypress
CYPRESS_IMAGE=my-org/cypress-with-plugins:latest \
  make -f .stoobly/services/Makefile test

Build those images however you like (e.g., start from mcr.microsoft.com/playwright or cypress/included and add packages), then reference them with the environment variable. Stoobly will use your custom image without touching the managed .Dockerfile.* files.

Q: How do I pass environment variables to my tests?

A: Add environment variables in the docker-compose.yml file under the environment section.

Example:

# .stoobly/services/entrypoint/test/docker-compose.yml
services:
  entrypoint.playwright:
    environment:
      - HTTP_PROXY=http://gateway:80
      - HTTPS_PROXY=http://gateway:80
      - BASE_URL=http://app.local
      - API_URL=http://api.local
      - [email protected]
      - TEST_USER_PASSWORD=testpass123
      - NODE_ENV=test

---

Running E2E Tests

Q: How do I record E2E test traffic?

A: Use the record workflow to capture all HTTP requests made during your E2E tests.

Example:

# Start record workflow
make -f .stoobly/services/Makefile record

# Tests run automatically in entrypoint container
# All HTTP requests are recorded

# View recorded requests
stoobly-agent request list

# Create scenario from recorded requests
stoobly-agent scenario create "E2E User Flow"

# Stop workflow
make -f .stoobly/services/Makefile record/down

Q: How do I run E2E tests with mocked responses?

A: Use the mock workflow to run tests against recorded responses without hitting real APIs.

Example:

# Ensure you have recorded responses
stoobly-agent scenario list

# Start mock workflow
make -f .stoobly/services/Makefile mock

# Tests run against mocked responses
# No real API calls are made

# Stop workflow
make -f .stoobly/services/Makefile mock/down

Q: How do I run E2E tests with response validation?

A: Use the test workflow to validate that responses match expected results.

Example:

# Start test workflow
make -f .stoobly/services/Makefile test

# Tests run with response validation
# Stoobly compares actual vs expected responses

# View test results
make -f .stoobly/services/Makefile test/logs

# Stop workflow
make -f .stoobly/services/Makefile test/down

Q: How do I run specific test files or suites?

A: Modify the command in docker-compose.yml to target specific tests.

Example for Playwright:

# Run specific test file
command: npx playwright test tests/login.spec.ts

# Run tests with specific tag
command: npx playwright test --grep @smoke

# Run tests in headed mode for debugging
command: npx playwright test --headed

Example for Cypress:

# Run specific spec
command: cypress run --spec "cypress/e2e/login.cy.js"

# Run with specific browser
command: cypress run --browser chrome

# Run with video recording
command: cypress run --video

---

Debugging

Q: How do I view test output and logs?

A: Use the logs command to see test execution output.

Example:

# View logs while tests are running
make -f .stoobly/services/Makefile test/logs

# View logs for specific service
make -f .stoobly/services/Makefile test/logs options="--service <SERVICE-NAME>"

# View logs for specific namespace
make -f .stoobly/services/Makefile test/logs options="--namespace <NAMESPACE>"

Q: How do I debug failing tests?

A: Enable headed mode and increase logging verbosity in the docker-compose.yml.

Example:

# .stoobly/services/entrypoint/test/docker-compose.yml
services:
  entrypoint.playwright:
    environment:
      - DEBUG=pw:api                    # Playwright debug logs
      - PWDEBUG=1                       # Playwright debug mode
    command: npx playwright test --headed --debug

Q: How do I save test artifacts (screenshots, videos)?

A: Mount an artifacts directory to persist test outputs and configure your test framework to save artifacts.

Example:

# .stoobly/services/entrypoint/test/docker-compose.yml
services:
  entrypoint.playwright:
    volumes:
      - ../../..:/home/stoobly/tests
      - ../../../test-results:/home/stoobly/tests/test-results  # Artifacts

// playwright.config.ts
export default defineConfig({
  use: {
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  outputDir: 'test-results',
});

---

Advanced Configuration

Q: How do I mount test fixtures and data?

A: Add volume mounts to share fixtures between host and container.

Example:

# .stoobly/services/entrypoint/test/docker-compose.yml
services:
  entrypoint.playwright:
    volumes:
      - ../../..:/home/stoobly/tests           # Project root
      - ../../../test-data:/home/stoobly/data  # Test data
      - ../../../fixtures:/home/stoobly/fixtures # Fixtures

Q: How do I handle dynamic data in E2E tests?

A: Use Stoobly's rewrite rules to normalize dynamic values in the configure script.

Example:

# .stoobly/services/entrypoint/test/configure
#!/bin/bash

# Replace dynamic user IDs with fixed test ID
stoobly-agent config rewrite set \
  --pattern "https://api.local/users/.*" \
  --method GET \
  --mode test \
  --type response_param \
  --name "userId" \
  --value "test-user-123"

# Replace timestamps
stoobly-agent config rewrite set \
  --pattern "https://api.local/.*" \
  --method GET --method POST \
  --mode test \
  --type response_param \
  --name "timestamp" \
  --value "2024-01-01T00:00:00Z"

---

Best Practices

Q: Should I record once and mock for all subsequent runs?

A: Yes, this is the recommended approach for fast, reliable E2E tests that don't depend on external APIs.

Example:

# Step 1: Record once (against real APIs)
make -f .stoobly/services/Makefile record

# Step 2: Create scenario from recordings
stoobly-agent scenario create "User Journey - Login to Checkout"

# Step 3: Snapshot for version control
stoobly-agent scenario snapshot user-journey --decode
git add .stoobly/snapshots/ && git commit -m "Add E2E test snapshots"

# Step 4: All future runs use mocks (fast, no external dependencies)
make -f .stoobly/services/Makefile mock

Q: How do I organize E2E tests by feature?

A: Create separate services or custom workflows for different test suites.

Example:

# Create service for auth tests
stoobly-agent scaffold service create auth-tests \
  --workflow test

# Create service for checkout tests
stoobly-agent scaffold service create checkout-tests \
  --workflow test

# Run specific test suite
make -f .stoobly/services/Makefile test STOOBLY_WORKFLOW_SERVICE_OPTIONS="--service auth-tests"

---

Complete Example

Q: What's a complete example of setting up Playwright E2E tests?

A: Here's a full example from scaffold to running tests.

Step 1: Scaffold the app

# Create app with Playwright
stoobly-agent scaffold app create my-e2e-app \
  --plugin playwright \
  --proxy-port 8080 \
  --ui-port 4200

# Add frontend service (test workflow only - serves static assets from fixtures)
stoobly-agent scaffold service create frontend \
  --hostname app.local \
  --port 80 \
  --scheme http \
  --workflow test

# Add API service (all workflows created by default)
# Remote upstream where hostname/port/scheme differ
stoobly-agent scaffold service create api \
  --hostname api.local \
  --port 443 \
  --scheme https \
  --upstream-hostname api.production.com

Step 2: Configure entrypoint docker-compose.yml

# .stoobly/services/entrypoint/test/docker-compose.yml
services:
  entrypoint.playwright:
    build:
      context: .
      dockerfile: .Dockerfile.playwright
      args:
        PLAYWRIGHT_IMAGE: mcr.microsoft.com/playwright:v1.40.0-jammy
        USER_ID: ${USER_ID}
    working_dir: /home/stoobly/app
    volumes:
      - ../../..:/home/stoobly/app
    depends_on:
      entrypoint.configure:
        condition: service_completed_successfully
    networks:
      app.ingress: {}
    environment:
      - HTTP_PROXY=http://gateway:80
      - HTTPS_PROXY=http://gateway:80
      - BASE_URL=http://app.local
      - API_URL=http://api.local
    profiles:
      - ${WORKFLOW_NAME}
    command: npx playwright test

Step 3: Configure Stoobly rules

# .stoobly/services/entrypoint/test/configure
#!/bin/bash

# Rewrite API calls
stoobly-agent config rewrite set \
  --pattern "https://api.production.com/.*" \
  --method GET --method POST --method PUT --method DELETE \
  --mode test \
  --hostname api.local

# Exclude analytics
stoobly-agent config firewall set \
  --pattern "https://.*analytics.com/.*" \
  --method GET --method POST \
  --mode test \
  --action exclude

Step 4: Create Playwright config

// playwright.config.ts
import { defineConfig } from '@playwright/test';

export default defineConfig({
  testDir: './tests',
  use: {
    baseURL: process.env.BASE_URL || 'http://app.local',
    proxy: {
      server: 'http://gateway:80',
    },
    ignoreHTTPSErrors: true,
    screenshot: 'only-on-failure',
    video: 'retain-on-failure',
  },
  outputDir: 'test-results',
});

Step 5: Create test

// tests/login.spec.ts
import { test, expect } from '@playwright/test';

test('user can login', async ({ page }) => {
  await page.goto('/login');
  
  await page.fill('[name="email"]', '[email protected]');
  await page.fill('[name="password"]', 'password123');
  await page.click('button[type="submit"]');
  
  await expect(page).toHaveURL('/dashboard');
  await expect(page.locator('h1')).toContainText('Welcome');
});

Step 6: Run tests

# Run with recording
make -f .stoobly/services/Makefile record

# Run with mocking
make -f .stoobly/services/Makefile mock

# Run with validation
make -f .stoobly/services/Makefile test

---

Quick Reference

Key files to modify for E2E testing:

.stoobly/services/entrypoint/test/
├── docker-compose.yml          # Add your test execution command
├── .Dockerfile.playwright      # Add dependencies (Playwright)
├── .Dockerfile.cypress         # Add dependencies (Cypress)
├── configure                   # Set Stoobly rules
└── init                        # Add initialization logic

Your project root:
├── playwright.config.ts        # Configure Playwright
├── cypress.config.js           # Configure Cypress
├── tests/                      # Your test files
└── package.json                # Test dependencies

Common commands:

# Create app with E2E plugin
stoobly-agent scaffold app create my-app --plugin playwright
stoobly-agent scaffold app create my-app --plugin cypress

# Run E2E tests
make -f .stoobly/services/Makefile record    # Record API calls
make -f .stoobly/services/Makefile mock      # Run with mocks
make -f .stoobly/services/Makefile test      # Run with validation

# View logs
make -f .stoobly/services/Makefile test/logs

# Stop workflow
make -f .stoobly/services/Makefile test/down