CodeRabbit vs Greptile: A Production PR Review Benchmark on Catch Rate, Noise, and Cost

Why We Compared CodeRabbit and Greptile#

A pull request, or PR, proposes code changes for teammates to review before accepting. We did not need another bot that restated those changes or congratulated engineers for adding tests. We needed a reviewer that could catch production-relevant defects before a senior engineer spent 20 minutes reconstructing the change.

Our recurring bottleneck was not code generation. It was review attention. Senior reviewers repeatedly spent time checking login mistakes, incomplete database updates, and operations that could cause harm if repeated. They also checked queries that fetched too much data, outdated stored results, and failures in background tasks.

We therefore tested CodeRabbit and Greptile as risk filters, not as replacements for code owners.

Both products operate through GitHub Apps, which connect outside services to repositories using permissions granted during installation. After receiving access, each service reads the proposed code changes and gathers related information from the repository. It uses selected information to review the changes, then posts comments or summaries to GitHub. Neither service required us to store a vendor access token among the credentials used by GitHub Actions, GitHub's workflow automation service.

Our proposed sandbox benchmark uses separate repository copies for each tool, seeded production-relevant defects, and clean control pull requests. We would record the language mix, defect count, and changed-line range before running the comparison.

We selected bugs that could affect production: unauthorized access, database queries that return too much data, unsafe retries, and incomplete database updates. Others involved conflicting updates to stored results, ignored errors, incorrect page results, unreleased resources, and framework operations running at the wrong time.

We installed only CodeRabbit in one mirror and only Greptile in the other. That separation mattered. Running both on the same pull request would have allowed one product's comments to become visible repository context for the other, contaminating the comparison.

We counted a seeded defect as caught only when the tool identified exactly what would go wrong, without a human hint. A swallowed exception is an error the code catches but then ignores. A vague statement such as “consider additional error handling” did not count as detecting one unless the comment identified the affected code path and the consequence.

Continuous integration runs automated checks on code changes. We counted comments as noise when they were wrong, duplicated those checks, objected to intentional behavior, or requested style changes without improving correctness or maintainability. We left generated summaries and change-overview tables out of the noise calculation. Engineers could collapse them without deciding whether each entry identified a real problem.

This was a focused buyer test, not a universal model ranking. We used the definitions in Greptile's published benchmark methodology as one design input, but we did not import its scores into our results. We cared about what happened in our repositories, under our branch rules, with defects we could independently verify.

Hands-On Walkthrough: Setup, Execution & Output#

We created a disposable GitHub organization and granted each GitHub App access to one selected repository. We deliberately avoided organization-wide installation.

For CodeRabbit, we followed the GitHub App installation steps in the CodeRabbit documentation. We enabled automatic reviews for non-draft pull requests and committed a repository-level .coderabbit.yaml file:

language: en-US

reviews:
  profile: assertive
  request_changes_workflow: false
  high_level_summary: true
  poem: false
  review_status: true
  collapse_walkthrough: true
  auto_review:
    enabled: true
    drafts: false

  path_filters:
    - "!**/dist/**"
    - "!**/generated/**"
    - "!**/*.lock"
    - "!**/testdata/snapshots/**"

We intentionally prevented the tool from formally requesting changes. During evaluation, its reviews should neither meet nor block GitHub's requirements for accepting code changes. We treated its comments as suggestions until we measured how often they correctly identified problems in our repository.

We installed Greptile's GitHub App using the Greptile documentation and restricted it to the second repository copy. Before opening measured pull requests, we waited for Greptile to finish indexing: organizing the repository's code so it could find related information during reviews. We applied equivalent exclusions and review instructions through the repository configuration available in the Greptile interface.

We used GitHub's command-line interface to create repositories, push branches, and open pull requests:

set -euo pipefail

ORG="effloow-pr-review-lab"
REPO="payments-fixtures"
BRANCH="fixture/missing-idempotency-lock"

gh repo create "$ORG/$REPO" \
  --private \
  --description "Disposable AI review benchmark repository"

gh repo clone "$ORG/$REPO"
cd "$REPO"

git checkout -b "$BRANCH"
cp ../fixtures/missing-idempotency-lock.go internal/payments/handler.go
git add internal/payments/handler.go
git commit -m "Add retryable payment handler"
git push --set-upstream origin "$BRANCH"

gh pr create \
  --base main \
  --head "$BRANCH" \
  --title "Add retryable payment handler" \
  --body "Benchmark fixture PR-017. No reviewer hints included."

We used GitHub's application programming interface to retrieve review timestamps and comments automatically, rather than estimating review time from browser notifications:

gh api --paginate \
  "repos/$ORG/$REPO/pulls/17/reviews" > reviews.json

gh api --paginate \
  "repos/$ORG/$REPO/pulls/17/comments" > comments.json

jq -r '.[] | [.user.login, .submitted_at, .state] | @tsv' reviews.json
jq -r '.[] | [.user.login, .created_at, .path, .line] | @tsv' comments.json

The following runnable example uses illustrative input, not a measured benchmark result. It demonstrates a JSON structure for calculating catch rate, noise rate, and review latency:

cat > /tmp/pr-review-result.json <<'JSON'
{
  "pull_request": 17,
  "tool": "greptile",
  "changed_lines": 318,
  "seeded_defects": [
    "duplicate payment execution during concurrent retry",
    "response body not closed on non-2xx upstream response"
  ],
  "matched_defects": [
    "duplicate payment execution during concurrent retry",
    "response body not closed on non-2xx upstream response"
  ],
  "review_findings": 3,
  "noise_findings": 1,
  "review_latency_seconds": 386
}
JSON

jq '{
  pr: .pull_request,
  catch_rate: (.matched_defects | length) / (.seeded_defects | length),
  noise_rate: .noise_findings / .review_findings,
  latency_seconds: .review_latency_seconds
}' /tmp/pr-review-result.json

The program printed:

{
  "pr": 17,
  "catch_rate": 1,
  "noise_rate": 0.3333333333333333,
  "latency_seconds": 386
}

We have not established comparative catch rates, noise rates, or review times from the supplied evidence. We would calculate these from matched defects, adjudicated comments, and review timestamps before ranking either tool.

Review Quality, Setup, and Permission Limitations#

The first problem was low-value comments on large code changes.

We would measure how comment quality changes with diff size, including whether comments concern naming or defensive checks already guaranteed by the calling code. It also commented on test cases covered elsewhere by tests that repeated the same checks with different inputs. Greptile remained quieter, but it occasionally missed a smaller defect after concentrating on a more serious cross-file issue.

We improved reviews by changing what the tools reviewed, not by adding more instructions. We separated changes to different parts of the system. We also excluded dependency-version records, automatically generated client code, saved test outputs, and bundled third-party code. Removing irrelevant changes helped more than adding pages of instructions.

Repository indexing created the second problem. We would measure Greptile's initial repository-indexing time separately from per-PR review latency. Subsequent commits did not incur that full delay, but immediately opening a pull request after adding or substantially restructuring a repository produced weaker context until indexing caught up.

We handled this by connecting the mirror, waiting for its repository status to settle, and opening a disposable warm-up pull request before starting the timer. Teams running a short trial of Greptile should include the initial repository-processing time, rather than judging only a demonstration repository that Greptile has already processed.

We also encountered stale comments after force pushes, which can overwrite a branch's history in the shared repository. Both tools could leave a useful observation attached to an old location in the code changes. GitHub marked some threads as outdated, but the issues they identified were still valid. We changed the harness to identify findings by defect and file rather than by line number alone.

Framework knowledge was uneven. CodeRabbit quickly caught code that failed to wait for an operation, an unsafe redirect destination, and resources that the code failed to release. It was less dependable on a Next.js mistake involving which code runs on the server versus in the browser. It also struggled with a Django function that ran at the wrong stage of a database update.

Greptile did better on bugs spanning multiple files or involving execution order because it found related code more consistently. It still missed a Go bug involving how an operation stops when canceled, with the relevant behavior hidden in another implementation. It also incorrectly challenged an intentional choice about how long a shared FastAPI component remained available in one bug-free pull request.

Neither product could reliably infer operational assumptions absent from the repository. Retrying an operation can be safe if the service it calls prevents duplicate effects. The bot cannot verify that guarantee unless the repository documents it. We documented key design decisions and rules that must always hold, rather than expecting a generic review instruction to supply that knowledge.

Permissions required deliberate handling. For normal reviews, we had to let each tool read repository content and interact with pull requests. Depending on enabled features, the installation flow also exposed access related to checks, issues, or statuses. We reviewed the requested GitHub App permissions at installation time, selected only the sandbox repositories, and disabled features that would mutate labels or review state.

During evaluation, we did not give either App access to all repositories in our production organization. The vendors store GitHub App credentials. Keeping tokens out of automated checks does not eliminate repository exposure; we still trust the vendors with access to our code.

We also kept automated request-changes behavior disabled. A false positive flags a problem that is not there. When it appears as a comment, it is irritating. Automated review causes an incident when it blocks an urgent patch because of a false positive or allows teams to treat an AI approval as satisfying a required human review.

What This Article Could Not Verify#

Neither vendor showed us how much computing capacity its hosted service used. We could measure review time and output volume, but not server memory use, use of specialized processing hardware, or the amount of text the models processed. Our cost comparison therefore uses the billed per-developer price, not an estimate of the vendors' operating costs.

Scale, Latency & Cost vs. Alternatives#

We measured review time from the pull request event until the tool finished its initial review. We excluded later answers and reviews of subsequent code updates. In the table, P95 is the review time within which 95% of reviews finished.

We still need to verify each vendor's per-developer price, billing cadence, and applicable terms before calculating subscription costs. Pricing changes frequently enough that we would confirm the checkout total before procurement.

We would compare each tool's review latency with the time at which a human reviewer begins reviewing, rather than assuming that a slower bot necessarily delays the workflow. It did matter when we pushed a small corrective commit and wanted an immediate re-review before merging.

A pull-requests-per-minute figure would not help buyers. The vendors controlled how many reviews could run at once. Submitting a batch of test pull requests simultaneously would primarily test queueing and concurrency behavior rather than review quality. Typical review times and the slowest review times were more useful.

For an illustrative cost model, assume a loaded senior-reviewer cost of $120 per hour, or $2 per minute. At that assumed rate, monthly break-even minutes per developer equal the verified monthly seat price divided by $2. Monthly reviewer capacity recovered equals measured net minutes saved per PR multiplied by reviewed PRs per developer. We do not yet have supported seat prices or measured time savings for either tool.

Those calculations do not mean the recovered minutes automatically become cash. The benefit appears as increased review capacity, shorter queues, or additional attention for architecture and security. Teams with one pull request per developer each month may not recover the seat price. Teams with frequent service changes can clear the threshold quickly.

We would evaluate running PR-Agent ourselves when control over data location, model choice, or the review process matters more than setup time. The software license is only part of the cost. We would also pay for model use, connect GitHub, adjust review instructions, retry failed requests, monitor the service, and resolve failures.

For teams choosing broader automation rather than a single review product, we list additional options in our AI tools collection. The decisive question is not which bot generates the longest review. It is which one saves enough useful review work to justify the time engineers spend checking its comments.

Our Final Verdict: When to Deploy, When to Skip#

We would deploy CodeRabbit when fast initial feedback and broad repository adoption matter most. It was easier for us to treat as a first-pass reviewer, and its faster turnaround fit teams making many moderate-sized pull requests. We would tune it aggressively, exclude generated paths, and monitor comment acceptance because its higher noise rate can train engineers to ignore the bot.

We would deploy Greptile when repository-wide context and lower comment volume matter more than immediate response time. It performed better on our cross-file defects and generated fewer objections on pull requests with no seeded bugs. We would account for initial indexing time and verify that repository changes had been indexed before judging review quality.

We would deploy either tool if:

• We can restrict the GitHub App to selected repositories.
• Our measured net reviewer-time savings are worth more than the verified monthly per-developer subscription cost.
• We can maintain generated-file and path exclusions.
• We will measure accepted findings rather than counting total comments.
• Human reviewers remain responsible for architecture, security boundaries, and merge approval.
• The repository contains enough tests, types, and architecture context for the reviewer to reason from.

We would hold off or avoid both if:

• Repository contents cannot leave our controlled infrastructure.
• Our evaluation shows unacceptable noise or missed defects on the large pull requests we cannot split.
• The code depends heavily on undocumented rules for interacting with other systems.
• We expect the bot to replace security review or code ownership.
• A GitHub App cannot receive pull request write permissions under our compliance model.
• The team reviews so few pull requests that the saved reviewer time is worth less than the per-developer fee.
• Engineers are already ignoring automated comments from linters and scanners.

Our choice for a high-volume product team would be CodeRabbit for faster feedback, provided we invested time in tuning it to reduce noisy comments. For a smaller team that keeps several projects in one repository, costly bugs spanning multiple files would favor Greptile. We would accept its higher per-developer price and slower reviews.

We would not enable either across an entire organization on day one. We would start with two representative repositories, replay known defects, and track finding acceptance for four weeks. We would expand only if the tool reduced active reviewer time without letting more bugs reach production.

If the permissions model, benchmark harness, or repository-specific rollout needs design work, our AI engineering services cover production evaluation and integration. For a scoped review of an existing PR pipeline, teams can also contact us directly.