I Replaced Manual QA with Claude Code Agents in My GitLab Pipeline. Here’s Exactly How.
The agent reads business requirements, analyzes the source code, discovers edge cases on its own, generates Karate test scenarios from scratch, runs them against a real database, commits the tests to the branch, and posts pass/fail results on the merge request. The developer reviews and merges.
If you’ve watched Iron Man, you’ve probably seen the moment where Tony Stark casually says, “JARVIS, run a diagnostic” and within seconds JARVIS scans everything, finds problems, and reports back.
Every time I sit waiting for tests to be written and run a pull request waits, I wish we had that.
Because in almost every enterprise Java project I’ve worked on, the same bottleneck appears.
The code gets written.
The pull request goes up.
And then… it just sits there.
Waiting for someone to write tests.
Waiting for someone to review.
Waiting for QA to verify edge cases that everyone knows about but nobody has time to cover :)
It’s not that engineers don’t care about testing. Everyone agrees tests are important. But writing good tests takes time, context, and patience. And in busy teams, that’s exactly what people are short on.
So I started thinking, what if we had something a bit closer to JARVIS?
Not another unit test generator that spits out mocked assertions nobody trusts.
But something that actually behaves like a QA engineer. Who reads the requirements, studies the code, thinks about what could go wrong, writes real end-to-end tests, runs them, and tells me what passed and failed. That was the idea which started this project.
Tech Stack: Java 21, Spring boot, PostgreSql, Flyway, GitLab CICD, Karate Tests, Claude Code
Core Concept: A developer build an application or bunch of features, but needs an automated way to test out the features that are being shipped before merging the code to the release branch. Claude code acts as an agentic qa engineer in the cicd pipeline, uses karate tests to define the behavior and run the real tests against an application endpoint to validate a set of outcomes and let’s the developer know about the results in the pull request.
The Application
The demo project is a Spring Boot 3.3 / Java 21 event aggregation service. Let me walk through the actual implementation so you can see what the agent analyzes.
Database Schema (Flyway V1)
CREATE TABLE events (
id BIGSERIAL PRIMARY KEY,
event_id VARCHAR(255) NOT NULL UNIQUE,
event_type VARCHAR(100) NOT NULL,
source_system VARCHAR(100),
payload JSONB NOT NULL,
received_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
processed BOOLEAN NOT NULL DEFAULT FALSE,
processed_at TIMESTAMPTZ,
batch_id VARCHAR(64)
);
CREATE TABLE event_snapshots (
id BIGSERIAL PRIMARY KEY,
event_id BIGINT NOT NULL REFERENCES events(id),
snapshot_data JSONB NOT NULL,
captured_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
batch_id VARCHAR(64) NOT NULL
);
CREATE TABLE job_executions (
id BIGSERIAL PRIMARY KEY,
job_name VARCHAR(100) NOT NULL,
last_run_at TIMESTAMPTZ NOT NULL,
status VARCHAR(20) NOT NULL DEFAULT ‘COMPLETED’,
batch_id VARCHAR(64) UNIQUE,
records_sent INTEGER NOT NULL DEFAULT 0,
error_message TEXT,
started_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
completed_at TIMESTAMPTZ
);
CREATE TABLE job_lock (
lock_name VARCHAR(100) PRIMARY KEY,
locked_by VARCHAR(255) NOT NULL,
locked_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ NOT NULL
);Four tables. The events table holds raw records with a processed flag. The event_snapshots table stores a copy of each event’s payload after dispatch, so the next run can calculate what changed. The job_executions table tracks every run. The job_lock table provides distributed locking via INSERT ON CONFLICT.
AggregationScheduler.java (The Orchestrator)
This is the class the agent spends the most time analyzing. Every branch in this code is a potential edge case.
@Component
@RequiredArgsConstructor
@Slf4j
public class AggregationScheduler {
private static final String LOCK_NAME = "aggregation-cron-lock";
private final AggregationProperties properties;
private final EventRepository eventRepository;
private final JobExecutionRepository jobExecutionRepository;
private final DiffCalculator diffCalculator;
private final PayloadAggregator payloadAggregator;
private final TargetApiClient targetApiClient;
private final DistributedLockService lockService;
@Scheduled(cron = "${app.aggregation.cron}")
public void scheduledRun() { execute(); }
public ExecutionResult execute() {
// ← Agent discovers: what if lock is already held?
if (!lockService.tryAcquire(LOCK_NAME))
return new ExecutionResult("SKIPPED", "Lock held by another instance", null, 0);
String batchId = UUID.randomUUID().toString();
JobExecutionEntity jobExec = null;
try {
Instant startedAt = Instant.now();
jobExec = jobExecutionRepository.save(JobExecutionEntity.builder()
.jobName(properties.jobName()).lastRunAt(startedAt).status("RUNNING")
.batchId(batchId).startedAt(startedAt).build());
// ← Agent discovers: orElse(Instant.EPOCH) means first-ever run
// fetches ALL events regardless of received_at
Instant lastRunAt = jobExecutionRepository
.findLastSuccessfulRun(properties.jobName())
.map(JobExecutionEntity::getLastRunAt)
.orElse(Instant.EPOCH);
List<EventEntity> events = lastRunAt.equals(Instant.EPOCH)
? eventRepository.findUnprocessedEvents(properties.batchSize())
: eventRepository.findUnprocessedEventsSince(lastRunAt, properties.batchSize());
// ← Agent discovers: empty DB = early return, no API call
if (events.isEmpty()) {
complete(jobExec, 0, null);
return new ExecutionResult("COMPLETED", "No unprocessed events", batchId, 0);
}
DiffSection diff = diffCalculator.calculateDiff(lastRunAt, events);
AggregatedPayloadRequest payload = payloadAggregator.buildPayload(batchId, events, diff);
TargetApiResponse response = targetApiClient.dispatch(payload);
// ← Agent discovers: markEventsProcessed and captureSnapshots
// ONLY run inside this success branch. On failure, events stay
// unprocessed and no snapshots are saved.
if (!"FAILED".equals(response.status())) {
markEventsProcessed(events, batchId);
diffCalculator.captureSnapshots(events, batchId);
complete(jobExec, events.size(), computeHash(payload.toString()));
return new ExecutionResult("COMPLETED", "Dispatched successfully", batchId, events.size());
} else {
fail(jobExec, response.message());
return new ExecutionResult("FAILED", response.message(), batchId, 0);
}
} catch (Exception e) {
if (jobExec != null) fail(jobExec, e.getMessage());
return new ExecutionResult("FAILED", e.getMessage(), batchId, 0);
} finally {
lockService.release(LOCK_NAME);
}
}
@Transactional
protected void markEventsProcessed(List<EventEntity> events, String batchId) {
eventRepository.markAsProcessed(
events.stream().map(EventEntity::getId).toList(), Instant.now(), batchId);
}
// ... complete(), fail(), computeHash() helper methods
public record ExecutionResult(String status, String message, String batchId, int recordsProcessed) {}
}
An agent reading this discovers at least six edge cases without being told: first-ever run (Instant.EPOCH branch), empty events (early return), lock contention (SKIPPED), API failure (events not marked processed), recovery on next run (failed events remain processed=false), and the fact that markEventsProcessed and captureSnapshots only happen inside the success branch.
DiffCalculator.java
This is where the diff logic lives. The agent reads this to discover edge cases around payload comparison
@Service
@RequiredArgsConstructor
@Slf4j
public class DiffCalculator {
private final EventSnapshotRepository snapshotRepository;
private final EventConverter eventConverter;
public DiffSection calculateDiff(Instant since, List<EventEntity> currentEvents) {
List<EventPayload> newEvents = new ArrayList<>();
List<ChangedEvent> changedEvents = new ArrayList<>();
for (EventEntity event : currentEvents) {
Optional<EventSnapshotEntity> lastSnapshot =
snapshotRepository.findLatestByEventId(event.getId());
if (lastSnapshot.isEmpty()) {
// ← Agent discovers: no snapshot = new event
newEvents.add(eventConverter.toEventPayload(event));
} else {
Map<String, Object> previous = lastSnapshot.get().getSnapshotData();
Map<String, Object> current = event.getPayload();
List<String> changedFields = detectChangedFields(previous, current);
// ← Agent discovers: if changedFields is empty, the event is
// EXCLUDED from both newEvents AND changedEvents.
// An unchanged event should not appear in the diff at all.
if (!changedFields.isEmpty()) {
changedEvents.add(new ChangedEvent(
event.getEventId(), current, previous, changedFields));
}
}
}
log.info("Diff: {} new, {} changed since {}", newEvents.size(), changedEvents.size(), since);
return new DiffSection(since, newEvents, changedEvents, newEvents.size(), changedEvents.size());
}
public void captureSnapshots(List<EventEntity> events, String batchId) {
List<EventSnapshotEntity> snapshots = events.stream()
.map(e -> EventSnapshotEntity.builder()
.eventId(e.getId()).snapshotData(e.getPayload())
.capturedAt(Instant.now()).batchId(batchId).build())
.toList();
snapshotRepository.saveAll(snapshots);
}
// ← Agent discovers: this compares ALL keys from both maps.
// A field added in current (not in previous) = changed.
// A field removed (in previous, not in current) = changed.
List<String> detectChangedFields(Map<String, Object> previous, Map<String, Object> current) {
Set<String> allKeys = new HashSet<>();
allKeys.addAll(previous.keySet());
allKeys.addAll(current.keySet());
return allKeys.stream()
.filter(k -> !Objects.equals(previous.get(k), current.get(k)))
.sorted()
.toList();
}
}
TargetApiClient.java
The agent reads this to understand failure handling and retry behavior
@Component
@RequiredArgsConstructor
@Slf4j
public class TargetApiClient {
private final WebClient targetApiWebClient;
private final TargetApiProperties targetApiProperties;
@Retry(name = "targetApi")
@CircuitBreaker(name = "targetApi", fallbackMethod = "fallbackDispatch")
public TargetApiResponse dispatch(AggregatedPayloadRequest payload) {
log.info("Dispatching batch [{}] ({} records)",
payload.batchId(), payload.totalRecords());
return targetApiWebClient.post()
.uri(targetApiProperties.endpoint())
.bodyValue(payload)
.retrieve()
.onStatus(HttpStatusCode::isError, resp ->
resp.bodyToMono(String.class)
.flatMap(body -> Mono.error(new RuntimeException(
"API error [" + resp.statusCode() + "]: " + body))))
.bodyToMono(TargetApiResponse.class)
.block();
}
// ← Agent discovers: when retries are exhausted or circuit breaker opens,
// this fallback returns status "FAILED". The scheduler checks for this
// and skips markEventsProcessed. So events remain unprocessed.
@SuppressWarnings("unused")
private TargetApiResponse fallbackDispatch(
AggregatedPayloadRequest payload, Throwable t) {
log.error("Retries exhausted for batch [{}]: {}", payload.batchId(), t.getMessage());
return new TargetApiResponse(
"FAILED", "Retries exhausted: " + t.getMessage(), 0, payload.batchId());
}
}AggregationController
@RestController
@RequestMapping("/api/v1/aggregation")
@RequiredArgsConstructor
public class AggregationController {
private final AggregationScheduler scheduler;
private final EventRepository eventRepository;
private final JobExecutionRepository jobExecutionRepository;
private final EventSnapshotRepository snapshotRepository;
@PostMapping("/trigger")
public ResponseEntity<ExecutionResult> trigger() {
ExecutionResult result = scheduler.execute();
return switch (result.status()) {
case "COMPLETED" -> ResponseEntity.ok(result);
case "SKIPPED" -> ResponseEntity.status(409).body(result);
case "FAILED" -> ResponseEntity.status(500).body(result);
default -> ResponseEntity.ok(result);
};
}
@GetMapping("/status")
public ResponseEntity<Map<String, Object>> status() {
long total = eventRepository.count();
long unprocessed = eventRepository.findUnprocessedEvents(Integer.MAX_VALUE).size();
return ResponseEntity.ok(Map.of(
"total_events", total,
"unprocessed_events", unprocessed,
"processed_events", total - unprocessed,
"total_snapshots", snapshotRepository.count(),
"total_job_executions", jobExecutionRepository.count()));
}
}The trigger returns {status, message, batchId, recordsProcessed} with HTTP 200/409/500. The status endpoint returns live counts. Both are what Karate tests assert against.
The Test Infrastructure
Who Writes What (Two-Layer Approach)
This is the design decision that makes the system work. There are two inputs, written by different people.
The product owner or tech lead writes REQUIREMENTS.md. This contains only business requirements. What the system should do. Not how to test it. Not what could go wrong. Just the functional specification.
## FR-1: Scheduled Aggregation with Manual Trigger
The service runs a scheduled cron job to aggregate unprocessed events
from the database and dispatch them to a downstream system.
POST /api/v1/aggregation/trigger allows manual triggering.
Only one instance should execute at a time across deployment.
## FR-3: Diff Calculation
Each payload must include a diff section showing what changed compared
to the previous run. Events never sent before appear as new. Events
whose payload changed appear as changed, including which fields differ.
## FR-5: Target API Dispatch
The payload is POSTed to a configurable endpoint with an X-API-Key header.
The service must handle transient failures with retries and protect
against sustained outages with a circuit breaker.Notice what’s missing, no edge cases. No “what if the database is empty” No “what if the API returns 500” No “what happens on the first ever run” The person writing requirements doesn’t need to think about those.
The Claude Code agent discovers edge cases by itself. It reads the requirements, then reads the source code, and reasons about what could go wrong. When it reads AggregationScheduler.java and sees this,
Instant lastRunAt = jobExecutionRepository
.findLastSuccessfulRun(properties.jobName())
.map(JobExecutionEntity::getLastRunAt)
.orElse(Instant.EPOCH); // ← First-ever run uses epoch
It realizes “What happens on the very first run when job_executions is empty? The code falls back to Instant.EPOCH, which means it fetches ALL unprocessed events regardless of received_at. I should test that”
When it reads TargetApiClient.java and sees the @CircuitBreaker annotation with a fallbackDispatch method that returns status: “FAILED”, it realizes “What happens to the events after a failed dispatch? Let me check the scheduler… ah, the markEventsProcessed only runs inside the if (!”FAILED”.equals(response.status())) block. So events should remain unprocessed. I should verify that.”
When it reads DiffCalculator.java and sees the snapshot comparison logic, it realizes: “What if an event’s payload hasn’t changed since the last snapshot? The detectChangedFields method would return an empty list, and the event would be excluded from changedEvents. But it would also not appear in newEvents because a snapshot exists. I should test that unchanged events are completely excluded from the diff.”
None of these edge cases are written anywhere. The agent discovers them by reading code. This is the whole point. A human QA engineer does the same thing: they read the spec, look at the code, and think “what if…?” The agent does that at machine speed, every merge request, without being asked.
The test lab is pre built. The test cases are not. Here’s what each piece does.
KarateTestRunner.java
This boots the entire stack. Read the properties carefully because they configure how the test environment works.
@SpringBootTest(
webEnvironment = SpringBootTest.WebEnvironment.DEFINED_PORT,
properties = {
"server.port=18090",
"app.aggregation.cron=-", // disable automatic scheduling
"app.aggregation.job-name=event-aggregation",
"app.aggregation.batch-size=100",
// Target API points BACK to this app's mock controller
"app.target-api.base-url=http://localhost:18090/mock/target",
"app.target-api.endpoint=/api/v1/ingest",
"app.target-api.timeout-seconds=5",
"app.target-api.api-key=test-api-key-12345",
// Disable retries in tests for predictable single-attempt behavior
"resilience4j.retry.instances.targetApi.max-attempts=1",
"resilience4j.circuitbreaker.instances.targetApi.sliding-window-size=100"
}
)
@ActiveProfiles("test")
@Testcontainers
public class KarateTestRunner {
@Container
static final PostgreSQLContainer<?> postgres =
new PostgreSQLContainer<>("postgres:16-alpine")
.withDatabaseName("aggregator_test")
.withUsername("test_user")
.withPassword("test_pass");
@DynamicPropertySource
static void props(DynamicPropertyRegistry r) {
r.add("spring.datasource.url", postgres::getJdbcUrl);
r.add("spring.datasource.username", postgres::getUsername);
r.add("spring.datasource.password", postgres::getPassword);
}
@Karate.Test
Karate testAll() {
// Pass runtime values to Karate via system properties
return Karate.run("classpath:karate/features")
.systemProperty("app.baseUrl", "http://localhost:18090")
.systemProperty("db.url", postgres.getJdbcUrl())
.systemProperty("db.username", postgres.getUsername())
.systemProperty("db.password", postgres.getPassword());
}
}
When this runs, Testcontainers starts a real PostgreSQL 16 container, Flyway applies the migration, and Spring Boot starts on port 18090. The target-api.base-url points to http://localhost:18090/mock/target, which is inside the same app. So when the scheduler dispatches a payload, it lands in the mock controller below.
MockTargetApiController.java
This is the fake “external system.” It runs inside the test app and captures every outbound payload so Karate can inspect what was sent.
@RestController
@RequestMapping("/mock/target")
@Profile("test")
public class MockTargetApiController {
private final List<JsonNode> receivedPayloads = new CopyOnWriteArrayList<>();
private int nextStatusCode = 200;
// ── The endpoint the scheduler actually calls ──
@PostMapping("/api/v1/ingest")
public ResponseEntity<String> ingest(@RequestBody JsonNode payload) {
receivedPayloads.add(payload); // CAPTURE every payload
if (nextStatusCode >= 400)
return ResponseEntity.status(nextStatusCode).body("{\"status\":\"ERROR\"}");
String batchId = payload.has("batch_id") ? payload.get("batch_id").asText() : "unknown";
int total = payload.has("total_records") ? payload.get("total_records").asInt() : 0;
return ResponseEntity.ok(String.format(
"{\"status\":\"OK\",\"message\":\"Accepted\",\"records_accepted\":%d,\"batch_id\":\"%s\"}",
total, batchId));
}
// ── Karate inspection endpoints ──
@GetMapping("/received")
public ResponseEntity<List<JsonNode>> getReceived() {
return ResponseEntity.ok(receivedPayloads);
}
@GetMapping("/received/last")
public ResponseEntity<JsonNode> getLastReceived() {
return receivedPayloads.isEmpty()
? ResponseEntity.notFound().build()
: ResponseEntity.ok(receivedPayloads.get(receivedPayloads.size() - 1));
}
@GetMapping("/received/count")
public ResponseEntity<Map<String, Integer>> getCount() {
return ResponseEntity.ok(Map.of("count", receivedPayloads.size()));
}
// ── Karate configuration endpoints ──
@PostMapping("/configure")
public ResponseEntity<String> configure(@RequestBody Map<String, Object> config) {
if (config.containsKey("statusCode"))
nextStatusCode = (int) config.get("statusCode");
return ResponseEntity.ok("configured");
}
@PostMapping("/reset")
public ResponseEntity<String> reset() {
receivedPayloads.clear();
nextStatusCode = 200;
return ResponseEntity.ok("reset");
}
}let me walk you through the actual code path. Look at the ingest method again carefully, there's an if check before the ok(). And the configure endpoint changes that field.
@PostMapping("/configure")
public ResponseEntity<String> configure(@RequestBody Map<String, Object> config) {
if (config.containsKey("statusCode"))
nextStatusCode = (int) config.get("statusCode"); // ← sets it to 500
return ResponseEntity.ok("configured");
}
So the timeline is:
1. App starts → nextStatusCode = 200 (default)
2. Karate calls: POST /mock/target/configure {statusCode: 500}
→ configure() sets nextStatusCode = 500
→ Returns ok("configured") — this ok() is just acknowledging the config change
3. Karate calls: POST /api/v1/aggregation/trigger
→ Scheduler runs → calls TargetApiClient.dispatch()
→ WebClient POSTs to /mock/target/api/v1/ingest
→ ingest() method runs:
receivedPayloads.add(payload); // captures the payload
if (500 >= 400) → TRUE // the check passes
return ResponseEntity.status(500) // returns HTTP 500, never reaches ok()
4. Karate calls: POST /mock/target/reset
→ nextStatusCode = 200 again (back to normal for the next scenario)DbUtils.java
A small Java helper that gives Karate feature files direct JDBC access to the test database.
public class DbUtils {
public static List<Map<String, Object>> query(Map<String, Object> config, String sql) {
List<Map<String, Object>> results = new ArrayList<>();
try (Connection c = getConn(config);
Statement s = c.createStatement();
ResultSet rs = s.executeQuery(sql)) {
ResultSetMetaData m = rs.getMetaData();
while (rs.next()) {
Map<String, Object> row = new LinkedHashMap<>();
for (int i = 1; i <= m.getColumnCount(); i++)
row.put(m.getColumnLabel(i), rs.getObject(i));
results.add(row);
}
} catch (Exception e) {
throw new RuntimeException("DB query failed: " + sql, e);
}
return results;
}
public static int execute(Map<String, Object> config, String sql) {
try (Connection c = getConn(config);
Statement s = c.createStatement()) {
return s.executeUpdate(sql);
} catch (Exception e) {
throw new RuntimeException("DB execute failed: " + sql, e);
}
}
private static Connection getConn(Map<String, Object> cfg) throws Exception {
Class.forName("org.postgresql.Driver");
return DriverManager.getConnection(
(String) cfg.get("url"), (String) cfg.get("username"), (String) cfg.get("password"));
}
}From a Karate feature file, this is how the agent’s generated tests insert data (setup) and verify database state (assertion)
* def db = Java.type('com.example.aggregator.karate.DbUtils')
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('evt-001', 'ORDER_CREATED', 'ecommerce', '{\"orderId\": \"ORD-100\", \"amount\": 249.99}', NOW() - INTERVAL '10 minutes')")
* def result = db.query(dbConfig, "SELECT count(*) as cnt FROM events WHERE processed = true AND batch_id IS NOT NULL")
* match result[0].cnt == 3Real SQL against a real PostgreSQL database. No mocking. No in-memory H2. The same queries you’d run in DBeaver to verify manually.
karate-config.js
This wires everything together for the feature files
function fn() {
var config = {
baseUrl: karate.properties['app.baseUrl'] || 'http://localhost:18090',
dbUrl: karate.properties['db.url'],
dbUsername: karate.properties['db.username'],
dbPassword: karate.properties['db.password']
};
// Derived URLs used in every scenario
config.triggerUrl = config.baseUrl + '/api/v1/aggregation/trigger';
config.statusUrl = config.baseUrl + '/api/v1/aggregation/status';
config.mockResetUrl = config.baseUrl + '/mock/target/reset';
config.mockLastUrl = config.baseUrl + '/mock/target/received/last';
config.mockCountUrl = config.baseUrl + '/mock/target/received/count';
config.mockConfigureUrl = config.baseUrl + '/mock/target/configure';
// JDBC config for DbUtils calls
config.dbConfig = {
url: config.dbUrl,
username: config.dbUsername,
password: config.dbPassword
};
return config;
}The agent reads this to understand which variables are available. When it generates * url triggerUrl, it knows that resolves to http://localhost:18090/api/v1/aggregation/trigger
The Empty Features Directory
src/test/resources/karate/features/ contains only .gitkeep. There are zero pre-written test scenarios. The agent creates everything.
Here’s what the agent generated after reading REQUIREMENTS.md and analyzing the source code. I'll walk through each scenario with the reasoning the agent would use to discover it.
What the Agent Generates
The agent creates three types of files, all committed to the branch:
1. Feature Files (Executable Tests)
Two .feature files in src/test/resources/karate/features/
aggregation-requirements.feature contains 6 scenarios (TC-01 through TC-06), one for each functional requirement. Each scenario follows the same pattern: insert data via SQL, trigger via HTTP, verify the captured payload, verify the database state.
aggregation-edge-cases.feature contains 9 scenarios (TC-07 through TC-15) that the agent discovered from reading the source code. Each has a comment explaining which class and code line led to the discovery. For example, below is feature file 1
Feature: Event Aggregation — Functional Requirements
Validates the core business requirements FR-1 through FR-6.
Background:
* def db = Java.type('com.example.aggregator.karate.DbUtils')
* db.execute(dbConfig, "DELETE FROM event_snapshots")
* db.execute(dbConfig, "DELETE FROM job_executions")
* db.execute(dbConfig, "DELETE FROM job_lock")
* db.execute(dbConfig, "DELETE FROM events")
* url mockResetUrl
* method post
* status 200
# ════════════════════════════════════════════════════════════════
# TC-01: Happy path — full aggregation flow [FR-1, FR-2, FR-4, FR-6]
#
# Validates: trigger endpoint works, events are read from DB,
# payload is constructed correctly, events are marked processed,
# snapshots are captured, job execution is recorded.
# ════════════════════════════════════════════════════════════════
Scenario: TC-01 Full aggregation flow — events processed and dispatched [FR-1, FR-2, FR-4, FR-6]
# Insert 3 events of different types
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('evt-001', 'ORDER_CREATED', 'ecommerce', '{\"orderId\": \"ORD-100\", \"amount\": 249.99}', NOW() - INTERVAL '10 minutes')")
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('evt-002', 'PAYMENT_RECEIVED', 'payments', '{\"paymentId\": \"PAY-200\", \"amount\": 249.99}', NOW() - INTERVAL '5 minutes')")
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('evt-003', 'INVENTORY_UPDATE', 'warehouse', '{\"sku\": \"SKU-789\", \"quantity\": 42}', NOW() - INTERVAL '3 minutes')")
# Verify 3 unprocessed events exist
* def before = db.query(dbConfig, "SELECT count(*) as cnt FROM events WHERE processed = false")
* match before[0].cnt == 3
# Trigger aggregation
* url triggerUrl
* method post
* status 200
* match response.status == 'COMPLETED'
* match response.recordsProcessed == 3
* match response.batchId == '#notnull'
* def batchId = response.batchId
# Verify mock received the payload
* url mockCountUrl
* method get
* match response.count == 1
# Inspect the actual payload sent
* url mockLastUrl
* method get
* status 200
* match response.batch_id == '#notnull'
* match response.timestamp == '#notnull'
* match response.total_records == 3
* match response.events == '#[3]'
* match each response.events contains { event_id: '#notnull', event_type: '#notnull', source_system: '#notnull', payload: '#notnull', received_at: '#notnull' }
# Verify diff section — all events are "new" on first run
* match response.diff == '#notnull'
* match response.diff.since == '#notnull'
* match response.diff.new_count == 3
* match response.diff.changed_count == 0
* match response.diff.new_events == '#[3]'
* match response.diff.changed_events == '#[0]'
# Verify specific event data integrity
* def order = karate.jsonPath(response, "$.events[?(@.event_id=='evt-001')]")[0]
* match order.event_type == 'ORDER_CREATED'
* match order.source_system == 'ecommerce'
* match order.payload.orderId == 'ORD-100'
* match order.payload.amount == 249.99
# Verify DB: all events marked processed
* def processed = db.query(dbConfig, "SELECT count(*) as cnt FROM events WHERE processed = true")
* match processed[0].cnt == 3
# Verify DB: processed_at timestamp is set
* def timestamps = db.query(dbConfig, "SELECT count(*) as cnt FROM events WHERE processed_at IS NOT NULL")
* match timestamps[0].cnt == 3
# Verify DB: batch_id matches the trigger response
* def batched = db.query(dbConfig, "SELECT count(*) as cnt FROM events WHERE batch_id = '" + batchId + "'")
* match batched[0].cnt == 3
# Verify DB: snapshots captured for future diff
* def snapshots = db.query(dbConfig, "SELECT count(*) as cnt FROM event_snapshots")
* match snapshots[0].cnt == 3
# Verify DB: job execution recorded as COMPLETED
* def jobs = db.query(dbConfig, "SELECT status, records_sent, batch_id FROM job_executions ORDER BY started_at DESC LIMIT 1")
* match jobs[0].status == 'COMPLETED'
* match jobs[0].records_sent == 3
* match jobs[0].batch_id == batchId
# ════════════════════════════════════════════════════════════════
# TC-02: Diff — new event on second run [FR-3]
#
# Validates: events sent for the first time appear in diff.new_events.
# On second run, a brand new event should be classified as "new."
# ════════════════════════════════════════════════════════════════
Scenario: TC-02 New event on second run appears in diff.new_events [FR-3]
# First run: one event
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('run1-001', 'ORDER_CREATED', 'ecommerce', '{\"orderId\": \"ORD-1\"}', NOW() - INTERVAL '20 minutes')")
* url triggerUrl
* method post
* status 200
* match response.recordsProcessed == 1
# Reset mock so we can inspect only the second run's payload
* url mockResetUrl
* method post
# Insert a brand new event
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('run2-001', 'ORDER_CREATED', 'ecommerce', '{\"orderId\": \"ORD-2\"}', NOW() - INTERVAL '1 minute')")
# Second run
* url triggerUrl
* method post
* status 200
* match response.recordsProcessed == 1
# Verify: new event appears in diff.new_events (no snapshot existed for it)
* url mockLastUrl
* method get
* match response.total_records == 1
* match response.diff.new_count == 1
* match response.diff.changed_count == 0
* match response.diff.new_events[0].event_id == 'run2-001'
# ════════════════════════════════════════════════════════════════
# TC-03: Diff — changed payload between runs [FR-3]
#
# Validates: when an event's payload changes between runs,
# it appears in diff.changed_events with previous/current values
# and the list of changed field names.
# ════════════════════════════════════════════════════════════════
Scenario: TC-03 Changed payload appears in diff.changed_events with field details [FR-3]
# First run: quantity = 50
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('diff-001', 'INVENTORY_UPDATE', 'warehouse', '{\"sku\": \"SKU-100\", \"quantity\": 50}', NOW() - INTERVAL '20 minutes')")
* url triggerUrl
* method post
* status 200
* match response.recordsProcessed == 1
# Reset mock
* url mockResetUrl
* method post
# Simulate payload change: quantity 50 → 30, mark unprocessed for reprocessing
* db.execute(dbConfig, "UPDATE events SET payload = '{\"sku\": \"SKU-100\", \"quantity\": 30}', processed = false, batch_id = null, processed_at = null WHERE event_id = 'diff-001'")
# Second run
* url triggerUrl
* method post
* status 200
* match response.recordsProcessed == 1
# Verify: changed event in diff with correct previous/current/fields
* url mockLastUrl
* method get
* match response.diff.changed_count == 1
* match response.diff.new_count == 0
* def changed = response.diff.changed_events[0]
* match changed.event_id == 'diff-001'
* match changed.current.quantity == 30
* match changed.previous.quantity == 50
* match changed.changed_fields contains 'quantity'
# sku didn't change, so it should NOT be in changed_fields
* match changed.changed_fields !contains 'sku'
# ════════════════════════════════════════════════════════════════
# TC-04: Target API failure — events remain unprocessed [FR-5, FR-6]
#
# Validates: on dispatch failure, events stay processed=false,
# no snapshots are captured, job is recorded as FAILED.
# ════════════════════════════════════════════════════════════════
Scenario: TC-04 Target API returns 500 — events not marked processed [FR-5, FR-6]
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('fail-001', 'ORDER_CREATED', 'ecommerce', '{\"orderId\": \"ORD-FAIL\"}', NOW() - INTERVAL '5 minutes')")
# Configure mock to return 500
* url mockConfigureUrl
* request { statusCode: 500 }
* method post
# Trigger — should fail
* url triggerUrl
* method post
* status 500
* match response.status == 'FAILED'
# Verify: event is still unprocessed
* def unprocessed = db.query(dbConfig, "SELECT processed, batch_id, processed_at FROM events WHERE event_id = 'fail-001'")
* match unprocessed[0].processed == false
* match unprocessed[0].batch_id == null
* match unprocessed[0].processed_at == null
# Verify: no snapshots captured
* def snapshots = db.query(dbConfig, "SELECT count(*) as cnt FROM event_snapshots")
* match snapshots[0].cnt == 0
# Verify: job execution recorded as FAILED with error message
* def jobs = db.query(dbConfig, "SELECT status, error_message, records_sent FROM job_executions ORDER BY started_at DESC LIMIT 1")
* match jobs[0].status == 'FAILED'
* match jobs[0].error_message == '#notnull'
* match jobs[0].records_sent == 0
# ════════════════════════════════════════════════════════════════
# TC-05: Payload JSON structure validation [FR-4]
#
# Validates: every field at every level matches the contract
# defined in FR-4. Types, nesting, required fields.
# ════════════════════════════════════════════════════════════════
Scenario: TC-05 Payload matches the JSON contract defined in FR-4 [FR-4]
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('schema-001', 'PAYMENT_RECEIVED', 'payments', '{\"paymentId\": \"PAY-500\", \"amount\": 150.75, \"currency\": \"USD\"}', NOW() - INTERVAL '5 minutes')")
* url triggerUrl
* method post
* status 200
* url mockLastUrl
* method get
* status 200
# Top-level structure
* match response == { batch_id: '#string', timestamp: '#string', total_records: '#number', events: '#array', diff: '#object' }
# batch_id is a UUID-like string (non-empty)
* match response.batch_id == '#regex [a-f0-9-]{36}'
# total_records matches events array length
* match response.total_records == karate.sizeOf(response.events)
# Event structure
* match response.events[0] == { event_id: '#string', event_type: '#string', source_system: '#string', payload: '#object', received_at: '#string' }
# Diff structure
* match response.diff contains { since: '#string', new_events: '#array', changed_events: '#array', new_count: '#number', changed_count: '#number' }
# Diff counts match array lengths
* match response.diff.new_count == karate.sizeOf(response.diff.new_events)
* match response.diff.changed_count == karate.sizeOf(response.diff.changed_events)
# Actual data values preserved correctly
* match response.events[0].payload.paymentId == 'PAY-500'
* match response.events[0].payload.amount == 150.75
* match response.events[0].payload.currency == 'USD'
# ════════════════════════════════════════════════════════════════
# TC-06: Status endpoint returns correct counts [FR-2]
#
# Validates: GET /status reflects actual DB state before and after.
# ════════════════════════════════════════════════════════════════
Scenario: TC-06 Status endpoint reflects correct state before and after [FR-2]
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('stat-001', 'ORDER_CREATED', 'ecommerce', '{\"orderId\": \"ORD-S1\"}', NOW() - INTERVAL '5 minutes')")
* db.execute(dbConfig, "INSERT INTO events (event_id, event_type, source_system, payload, received_at) VALUES ('stat-002', 'ORDER_CREATED', 'ecommerce', '{\"orderId\": \"ORD-S2\"}', NOW() - INTERVAL '3 minutes')")
# Check BEFORE trigger
* url statusUrl
* method get
* status 200
* match response.total_events == 2
* match response.unprocessed_events == 2
* match response.processed_events == 0
* match response.total_snapshots == 0
* match response.total_job_executions == 0
# Trigger
* url triggerUrl
* method post
* status 200
# Check AFTER trigger
* url statusUrl
* method get
* status 200
* match response.total_events == 2
* match response.unprocessed_events == 0
* match response.processed_events == 2
* match response.total_snapshots == 2
* match response.total_job_executions == 12. TEST-CASES.md (Human Readable Test Plan)
The agent also generates src/test/resources/karate/TEST-CASES.md, which serves as a living test plan document. It contains, A summary table showing total/passed/failed and how many came from requirements vs discovered.
| Metric | Count |
|--------|-------|
| Total test cases | 15 |
| From requirements | 6 |
| Edge cases discovered | 9 |
| Passed | 15 |
| Failed | 0 |A requirement coverage matrix so you can see at a glance which requirements are tested.
| Requirement | Description | Test Cases | Covered |
|-------------|-------------|------------|---------|
| FR-1 | Scheduled aggregation with manual trigger | TC-01 | Yes |
| FR-3 | Diff calculation | TC-02, TC-03 | Yes |
| FR-5 | Target API dispatch | TC-04 | Yes |And for each test case, a detailed entry with pre-conditions, steps, and expected results. Edge case entries include a “Discovered in” section that traces back to the exact Java code
### TC-14: No snapshots after failed dispatch [EDGE CASE]
**Discovered in:** `AggregationScheduler.java`:
if (!"FAILED".equals(response.status())) {
markEventsProcessed(events, batchId);
diffCalculator.captureSnapshots(events, batchId); // ← inside success
}
**Reasoning:** captureSnapshots is inside the success branch. After a failed
dispatch, no snapshots are saved. The next successful run should treat the
event as "new" in the diff (no prior snapshot exists).
**Pre-conditions:** 1 event. First run fails (mock 500). Second run succeeds.
**Expected:** After failure: 0 snapshots. After success: event in diff.new_events.
**Result:** PASSThis TEST-CASES.md becomes a living document in the repository. New developers read it to understand what's tested and why. When the product owner asks "is the diff logic tested?" you can point them to TC-02, TC-03, TC-09, TC-12, and TC-13.
3. The Agent’s MR Comment (Report)
Separately from the committed files, the agent posts a structured comment on the merge request with the pass/fail summary and edge case table. The developer sees this without opening any files
The CI/CD Pipeline Step by Step
This is where the magic cooks.
The .gitlab-ci.yml has three stages. Let me walk through each one.
Stage 1: build
Standard Gradle compile. Produces the JAR artifact.
build:
stage: build
image: eclipse-temurin:21-jdk
script:
- chmod +x gradlew
- ./gradlew clean assemble --no-daemonStage 2: claude-qa (The Agent Stage)
This is where everything happens. Here’s the flow,
┌─────────────────────────────────────────────────────────-┐
│ CI Runner (Docker container: node:20) │
│ │
│ before_script: │
│ [1] Install JDK 21 (agent needs to compile + run) │
│ [2] npm install -g @anthropic-ai/claude-code │
│ [3] git config with GITLAB_TOKEN (for pushing back) │
│ [4] git checkout feature-branch │
│ │
│ script: │
│ [5] claude --max-turns 30 "<prompt>" │
│ │ │
│ ├── Reads REQUIREMENTS.md (6 business FRs) │
│ ├── Reads CLAUDE.md (test scaffolding docs) │
│ ├── Scans src/main/ Java classes │
│ ├── DISCOVERS edge cases from code analysis │
│ ├── GENERATES .feature files (15 scenarios) │
│ ├── GENERATES TEST-CASES.md (test plan) │
│ ├── Runs ./gradlew test --tests '*Karate*' │
│ │ → Testcontainers spins up PostgreSQL │
│ │ → Spring Boot starts on port 18090 │
│ │ → Karate executes against live app + DB │
│ ├── Fixes failures → re-runs (up to 3x) │
│ └── Outputs structured report │
│ │
│ [6] git add .feature files + TEST-CASES.md │
│ git commit + push to feature branch │
│ → Files appear in MR diff for review │
│ │
│ [7] curl → POST report as MR comment via GitLab API │
│ → Developer sees pass/fail in the MR │
└─────────────────────────────────────────────────────────-┘How does authentication work?
This is the key thing. The Claude Code CLI checks for the ANTHROPIC_API_KEY environment variable. If it's set, the CLI uses it directly for API calls. No browser OAuth, no /login command, no interactive prompts.
# In GitLab: Settings → CI/CD → Variables
# Variable: ANTHROPIC_API_KEY
# Value: sk-ant-api03-xxxxx (your Anthropic API key)
# Masked: Yes
# Protected: Yes
# In .gitlab-ci.yml the variable is injected automatically:
variables:
ANTHROPIC_API_KEY: "${ANTHROPIC_API_KEY}"Claude Code CLI is just a Node.js program that:
1. Reads files from your repo (via Read tool)
2. Creates/edits files (via Write/Edit tools)
3. Runs shell commands (via Bash tool) │
4. Talks to Anthropic API for the AI reasoning
Let me explain the key parts of the cicd YAML.
before_script sets up the environment. The CI runner is node:20 (Claude Code CLI is an npm package). We install JDK 21 inside it because Gradle needs a JVM to compile Karate tests. Git is configured with GITLAB_TOKEN so the agent can push generated files back to the branch
before_script:
- apt-get update && apt-get install -y wget git
- wget -q <adoptium-jdk-url> -O /tmp/jdk.tar.gz
- mkdir -p /opt/java && tar xzf /tmp/jdk.tar.gz -C /opt/java
- npm install -g @anthropic-ai/claude-code
- git config user.email "claude-qa-agent@ci.local"
- git config user.name "Claude QA Agent"
- git remote set-url origin "https://oauth2:${GITLAB_TOKEN}@..."
- git checkout "${CI_COMMIT_REF_NAME}"The agent prompt is the core of the system. It tells the agent to read requirements, analyze code, discover edge cases, generate both .feature files and TEST-CASES.md, run tests, fix failures, and output a report.
The agent invocation is just ‘’Run the qa-agent workflow. Follow every step defined in theqa-agent agent definition, because the full prompt lives in .claude/agents/qa-agent.md
claude -p \
--permission-mode acceptEdits \
--allowedTools "Bash(./gradlew*) Bash(find*) Bash(cat*) Bash(ls*) Bash(grep*) Read Write Edit" \
"Run the qa-agent workflow. Follow every step defined in the
qa-agent agent definition (.claude/agents/qa-agent.md)..." \
2>&1 | tee claude-qa-report.txtClaude Code loads .claude/agents/qa-agent.md at session start (it reads the YAML frontmatter for tools, model, and maxTurns, and the markdown body for the full 7-step prompt). The pipeline just tells Claude to execute the workflow. All the intelligence is in the agent file
---
name: qa-agent
description: >
Senior QA automation agent that reads business requirements...
tools:
- Bash
- Read
- Write
- Edit
model: opus
maxTurns: 30
---
.......
## Step 0: Check If Tests Already Exist
Run:
find src/test/resources/karate/features -name "*.feature" | head -20
If .feature files already exist:
- Do NOT regenerate from scratch
- Read existing files and TEST-CASES.md
- Check if NEW requirements were added that aren't covered
- Check if implementation changed in ways that need new edge cases
- Only ADD new scenarios for uncovered items
- Append to existing files, don't overwrite
- Re-run all tests (existing + new) and fix any failures
If no .feature files exist:
- Proceed with full generation from Step 1Committing generated files to the branch is what makes the tests reviewable.
- |
git add src/test/resources/karate/features/*.feature
git add src/test/resources/karate/TEST-CASES.md
git commit -m "Claude QA Agent: generated Karate tests + test plan
[skip ci]"
git push origin "${CI_COMMIT_REF_NAME}"Posting the report to the MR uses the GitLab API.
- |
curl --request POST \
--header "PRIVATE-TOKEN: ${GITLAB_TOKEN}" \
--data "..." \
"${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/notes
Stage 3: deploy
Manual gate on the main branch. Customize for your deployment target.
Full gitlab-cicd.yaml and the qa-agent can be found here: https://l0xvfiqr5t8q8arl.public.blob.vercel-storage.com/gitlab-ci.yml
https://l0xvfiqr5t8q8arl.public.blob.vercel-storage.com/qa-agent.md (.claude/agents/qa-agent.md)
What the Developer Sees in the MR
When the pipeline finishes, the merge request shows below
- Pipeline badge: Green check or red X
- Diff tab: Three new files: aggregation-requirements.feature, aggregation-edge-cases.feature, and TEST-CASES.md. The developer reads TEST-CASES.md first for the overview, then inspects .feature files for details.
- Comment: The agent’s structured report with requirements coverage and discovered edge cases
What I Learned
Edge case discovery is the most valuable part. The agent consistently finds 8 to 12 edge cases from 6 business requirements. Some are obvious (empty data, API failure). Others are subtle (unchanged payload exclusion from diff, no snapshots after failed dispatch, added/removed field detection). A human QA engineer would find most of these too, but the agent does it in 3 minutes.
TEST-CASES.md changes how the team uses the test suite. Before this, tests were just code that passed or failed. Now there’s a readable document that maps every test to a requirement or code-level edge case, explains why it exists, and shows whether it passed. Product owners actually read it.
The two-layer separation works. Product owners write what the system does. The agent figures out what could go wrong. Nobody maintains an edge case list. When the code changes, the agent discovers new edge cases automatically.
Committing generated tests to the branch is critical. They become part of the codebase, run on every future pipeline, and are reviewable in the MR diff. This is fundamentally different from “generate, show results, throw away.”
DB verification catches real bugs. In one run, the agent’s SELECT count(*) FROM events WHERE processed = true assertion caught a case where events were being marked processed even after a failed API dispatch. Unit tests with mocked repositories would never have found that.
Cost and Performance
The claude-qa stage typically takes 5 to 8 minutes and uses 150K to 250K tokens, costing roughly $0.75 to $2.00 per run if you use a pay as you go model. Yet still its much lower than a QA engineer spends time on understanding the requirements, identifying tests, writing tests and creating a results report.