Building an AI Code Review Tool That Actually Catches Hallucinations
Discover how AI code review tools detect LLM hallucinations and generated vulnerabilities. Best practices for automating security checks on Copilot and Cha
Why Standard Code Review Fails on AI-Generated Code
Your team's code review process works fine for handwritten code. Developers write a function, explain their logic, reviewers ask questions, vulnerabilities surface through conversation. That entire model breaks when an LLM generates 500 lines of code in 10 seconds.
A modern AI code review tool needs to understand not just what the code does, but what assumptions the LLM made that might be wrong. Standard linters miss these because hallucinations aren't syntax errors. They're logic errors, false security assumptions, and impossible edge cases.
The Hallucination Problem
When you ask an LLM to "write a secure password hash function," it generates something like:
import hashlib
def hash_password(password):
# Hallucination: Assumes bcrypt is faster than it actually is
return hashlib.sha256(password.encode()).hexdigest()
This code:
- Compiles without errors
- Passes basic unit tests
- Passes static analysis (syntactically correct)
- Violates security best practices (SHA256 for passwords is wrong)
The LLM didn't make a typo. It made a fundamentally wrong security decision because it was trained on incomplete or outdated examples. Traditional code review tools can't catch this.
How AI Code Review Tools Work Differently
An effective AI code review tool uses multiple layers:
Layer 1: Semantic Analysis
Instead of just checking syntax, analyze what the code claims to do versus what it actually does:
# Pattern: Hashing function that doesn't use iterative hash
if "hash" in function_name and "password" in function_name:
if "hashlib" in imports and "sha" in code:
flag_security_issue("Password hashing uses non-iterative algorithm")
if "bcrypt" not in imports and "argon" not in imports:
flag_security_issue("Missing strong password hashing library")
Layer 2: Common LLM Hallucination Patterns
Train the tool to recognize mistakes LLMs make repeatedly:
HALLUCINATION_PATTERNS = [
# Pattern: Auth tokens without expiration
{
"name": "missing_token_expiration",
"regex": r"jwt\.encode.*verify=False",
"severity": "critical"
},
# Pattern: SQL without parameterization
{
"name": "sql_injection_risk",
"regex": r"execute\(f['\"].*{.*}.*['\"]\)",
"severity": "critical"
},
# Pattern: Hardcoded secrets
{
"name": "hardcoded_secrets",
"regex": r"(api_key|password|secret)\s*=\s*['\"](?!\$|{{).+['\"]" ,
"severity": "high"
}
]
Layer 3: Dependency Chain Validation
The AI code review tool checks not just the code, but every package it imports:
const auditDependencies = async (imports) => {
for (const lib of imports) {
const maintainer = await getNpmMaintainer(lib);
const lastUpdate = await getLastUpdate(lib);
// Flag unmaintained or single-author packages
if (daysSinceUpdate(lastUpdate) > 180) {
flag("Dependency appears unmaintained");
}
if (maintainer.accountRisk === "high") {
flag("Maintainer account shows suspension risk");
}
}
};
Building Your Own AI Code Review Tool
You don't need to buy enterprise software. A basic in-house AI code review tool can be built in a weekend:
from ast import parse, walk, Call, Name, Constant
import subprocess
import json
class AICodeReviewer:
def __init__(self):
self.findings = []
def review_python_file(self, filepath):
with open(filepath) as f:
tree = parse(f.read())
# Check 1: No MD5/SHA1 used for passwords
for node in walk(tree):
if isinstance(node, Call):
if hasattr(node.func, 'attr'):
if node.func.attr in ['sha1', 'md5']:
if 'password' in filepath.lower():
self.findings.append({
"severity": "high",
"message": "Weak hash algorithm used for passwords"
})
# Check 2: SQL query building
with open(filepath) as f:
content = f.read()
if 'execute' in content and '.format(' in content:
if 'SELECT' in content or 'INSERT' in content:
self.findings.append({
"severity": "critical",
"message": "Potential SQL injection via string formatting"
})
return self.findings
# Usage
reviewer = AICodeReviewer()
reviewer.review_python_file('auth.py')
print(json.dumps(reviewer.findings, indent=2))
This tool:
- Parses code into an abstract syntax tree (AST)
- Walks the tree looking for dangerous patterns
- Checks imports against a vulnerability database
- Outputs findings as structured data
Integration with Your Workflow
The AI code review tool works best as a CI/CD gate:
# .github/workflows/security.yml
name: AI Code Review
on: [pull_request]
jobs:
review:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Run AI code review
run: |
python ai_code_reviewer.py --input . --output findings.json
- name: Fail on critical issues
run: |
CRITICAL=$(jq '[.[] | select(.severity == "critical")] | length' findings.json)
if [ $CRITICAL -gt 0 ]; then
echo "Found $CRITICAL critical security issues"
exit 1
fi
Now every LLM-generated PR is scanned before human review. The tool catches the obvious hallucinations, your team reviews the subtle ones.
Key Takeaways
- LLM hallucinations aren't syntax errors, so standard code review tools miss them
- Effective AI code review tools use semantic analysis and LLM-specific pattern detection
- Automate dependency audits and common vulnerability patterns to catch what human review will skip
- Deploy AI code review as a CI/CD gate to prevent hallucinated code from reaching production