CasePilot for Jira

Documentation — how to install, configure, and get the most out of CasePilot.

How it works

CasePilot appears as a panel inside any Jira issue. It reads the issue title, description, acceptance criteria, and comments, then runs a three-pass AI pipeline to produce structured test cases.

Step 1 — Generating test cases

Generates an initial set of test cases covering positive, negative, and edge scenarios based on the full issue context. Takes about 10–15 seconds.

Step 2 — Reviewing and polishing

A second AI pass reviews the first output against the original acceptance criteria: fills coverage gaps, removes duplicates, improves step clarity, and scores overall quality 1–10.

Step 3 — Finalizing

Detects shared preconditions and overlapping coverage across the test set. Suggests groupings and potential merges without modifying the tests.

Typical run takes 30–40 seconds for the default batch of 5 test cases. Larger batches (10 tests) can take up to 60 seconds. A quality score and coverage gaps are shown after each generation.

Generating test cases

Open any Jira issue (Story, Task, Bug).
Scroll down to the CasePilot — AI Test Cases panel.
Select an output format: Test Cases, BDD / Gherkin, or Code.
Optionally choose a Strategy template and adjust the Max count.
Click Generate.
Review the results. Select which test cases to keep.
Click Create as Sub-tasks (or push to Xray / Zephyr Scale if configured).

CasePilot automatically reads existing CasePilot sub-tasks on the issue and avoids generating duplicates of already-created tests.

Requirements Quality & Deep Analysis

CasePilot automatically evaluates the quality of your issue before you generate test cases. A Requirements Quality Score (1–10) appears at the top of the panel, based on an instant heuristic check of the issue's title, description, and acceptance criteria.

Instant heuristic score

Displayed automatically when you open the CasePilot panel. Shows either "Ready to generate" or "Improve story for better results" depending on the score. No credits consumed — runs locally.

Deep Analysis (AI)

Click the Deep Analysis link next to the score to run an AI-powered detailed review. Returns:

Acceptance criteria breakdown with quality indicators per criterion
List of testability issues found in the story
Suggested improved acceptance criteria with click-to-copy

Deep Analysis uses 1 credit from your monthly quota. The heuristic score is always free.

Output formats

Test Cases (default)

Structured test cases with title, category (positive / negative / edge), priority (1–4), preconditions, numbered steps with expected results, test data, and optional automation hints.

BDD / Gherkin

Given / When / Then scenarios. Supports Scenario Outline with Examples tables for parameterized tests. Saved as sub-tasks with the full Gherkin text in the description.

Code (Playwright / Cypress / Selenium)

Copy-paste-ready test code generated for your chosen framework. Select the framework before generating — the output uses framework-specific syntax, locators, and assertions. Code is displayed in the panel with a Copy button; it is not saved as a sub-task.

Strategy templates

Strategy templates guide the AI on what kind of tests to generate and pre-set the Max count and categories.

Strategy	Focus	Default max
Comprehensive	Full coverage — positive, negative, edge	8
Smoke	Critical happy-path only	3
Regression	Areas likely to break from code changes	10
Security	OWASP Top 10 — auth, injection, access control	8
Performance	Load, concurrency, timeouts, boundary values	6
Accessibility	Keyboard navigation, screen readers, ARIA, contrast	6

Strategies are not available for Code format — use the framework selector instead.

Project Knowledge

Project Knowledge teaches CasePilot about your tech stack, UI components, business rules, and testing conventions. With it, generated tests reference your actual technology and follow your project's naming patterns instead of producing generic output.

To configure:

Go to Jira Settings → Apps → CasePilot for Jira.
Fill in any combination of: Tech Stack, UI Components, Business Rules, Testing Conventions.
Click Save. Settings apply to all future generations in this project.

All fields are optional. CasePilot works without Project Knowledge — it simply generates more generic output.

Confluence integration

Link Confluence pages as additional AI context. Useful for product requirements docs, API specs, or domain glossaries. CasePilot reads the page content and includes it when generating test cases. Up to 10 pages per issue.

Confluence is optional. CasePilot generates test cases from the issue description and acceptance criteria on its own — the Confluence step only adds extra context when you have pages worth including.

Project-level pages (applied to all issues)

Go to Jira Settings → Apps → CasePilot for Jira.
In the Confluence Pages section, select your Confluence space.
Search for and add specific pages. Click the × next to any selected page to remove it.
Click Save. The selected pages are used as AI context on every generation in this project.

Per-issue pages (override the project list)

Open any issue's CasePilot panel, expand Confluence Pages, and add or remove pages specifically for that issue.

By default, an issue inherits the project-level pages.
Adding or removing any page on the issue switches it to its own custom list.
Removing all pages persists as "no Confluence context for this issue" — the project defaults do not return automatically. To reinstate them, click Use project pages and confirm.
A small badge in the panel header shows how many pages are currently linked.

If Confluence isn't available on your site, CasePilot shows a note explaining that this step can be skipped. Test case generation keeps working normally without Confluence.

Only pages within your connected Confluence instance are accessible. CasePilot reads page text content only — attachments and embedded images are not processed. Large pages are truncated to fit the AI context.

Sprint Coverage Radar

Analyze test coverage across an entire sprint. Sprint Coverage Radar checks which issues have CasePilot-generated test cases (sub-tasks with the casepilot label) and highlights gaps and risks.

Go to Jira Settings → Apps → CasePilot for Jira and open the Tools tab.
Select a sprint from the dropdown.
Click Analyze.

The results show:

Overall coverage percentage for the sprint
Covered and uncovered issue counts
Risk level per issue (based on complexity and missing coverage)
Each issue is clickable — navigates directly to the issue in Jira

Sprint Coverage Radar is a heuristic analysis — it runs instantly, consumes no credits, and is available on all plans including Free.

Quality Time Machine

An ROI dashboard that tracks value delivered by CasePilot over time. Available in Jira Settings → Apps → CasePilot for Jira → Tools tab.

The dashboard displays:

Total test cases generated
Estimated hours saved
Estimated cost savings
Average quality score across generations

Data accumulates automatically as you generate test cases. No configuration needed — open the Tools tab to see your project's metrics.

Xray & Zephyr Scale integration

By default, CasePilot saves test cases as Jira sub-tasks. You can configure it to create tests directly in Xray or Zephyr Scale instead.

Xray setup

Generate Xray API credentials in your Xray settings (Client ID + Client Secret).
Go to Jira Settings → Apps → CasePilot for Jira → TMS Integration.
Select Xray, enter Client ID and Client Secret, click Test Connection.
Save.

Zephyr Scale setup

Generate a Zephyr Scale API token in your Zephyr settings.
Go to Jira Settings → Apps → CasePilot for Jira → TMS Integration.
Select Zephyr Scale, enter the API token, click Test Connection.
Save.

If TMS credentials are missing or the connection fails, CasePilot automatically falls back to creating Jira sub-tasks so no test cases are lost.

Plans and billing

CasePilot is billed per test case generated (not per test case saved). The count resets on the first day of each calendar month.

Plan	Test cases / month	Price
Free	20	$0
Starter	200	$35 / month
Pro	1,000	$79 / month
Team	3,000	$149 / month

Annual plans available at a discount (~17% off). All plans include unlimited users — the quota is per Jira project (org), not per user.

To upgrade: go to Jira Settings → Apps → CasePilot for Jira and click Upgrade Plan. Billing is managed through Atlassian Marketplace.

FAQ

Does CasePilot work on any issue type?

Yes — Stories, Tasks, and Bugs. It works best on issues with a description and acceptance criteria. Issues with no description will produce generic test cases.

Where is my issue content sent?

Issue title, description, and acceptance criteria are sent to the CasePilot API (hosted on Microsoft Azure, West Europe) and processed by OpenAI GPT-4o mini. Content is not stored after the result is returned.

Can multiple team members use the same quota?

Yes. The quota is shared across all users in the same Jira project. There is no per-seat limit.

What happens when I hit the limit?

Generation is blocked until the quota resets on the 1st of next month, or until you upgrade your plan. Already-created sub-tasks are not affected.

Does Code format count against my quota?

Yes — each generation (regardless of format) counts the number of test items produced against your monthly quota.

Can I regenerate without creating duplicates?

CasePilot checks for existing sub-tasks with the casepilot label on the issue and passes their titles to the AI to avoid generating the same tests again.

Why does generation take 30–60 seconds?

CasePilot runs three sequential AI passes — generating, reviewing, then finalizing. Each pass is a separate LLM call. The extra time produces significantly higher quality than single-pass tools. Small batches (3 tests) finish in ~20s; larger batches (10 tests) can take up to 60s.

What happens if I remove all Confluence pages from an issue?

The empty state is saved as an explicit "no Confluence context" override for that issue, so the project-default pages will not silently return on refresh. To re-link the project defaults for this issue, click Use project pages in the Confluence Pages section and confirm.

What if my Atlassian site doesn't have Confluence?

CasePilot detects this and tells you Confluence isn't available on this site. Test case generation keeps working — Confluence context is optional, not required.

Support

Questions not covered here? Contact us at [email protected] or visit our support page.