Skip to content
Logo
Published on

Screenshot-Grounded AI Development with Peekaboo

How Peekaboo screenshots provide the missing feedback loop for AI pair programmers, turning visual diffs into actionable guidance and helping agents make precise UI tweaks.

The Screenshot Gap in AI-Assisted Development

AI agents move fast. They can iterate through UI implementations, refactor components, and generate code at impressive speeds. But there's a fundamental problem: agents can't see what they're building.

When an AI suggests a CSS tweak or proposes a layout change, it's working blind. The agent understands the code structure, can reason about flex containers and grid layouts, but it can't actually see whether that margin-top: 20px creates visual balance or pushes content off-screen.

This is where screenshots become the missing feedback loop for AI pair programmers.

Enter Peekaboo v3

Peekaboo v3 is a macOS screenshot automation tool that integrates seamlessly with iOS simulators and native apps. Unlike traditional screenshot tools, Peekaboo provides:

  • Deterministic snapshots: Consistent output across runs
  • Rich metadata: JSON files with dimensions, scale factors, and more
  • Scriptable CLI: Easy integration with existing workflows
  • Predictable output: Files land in known directories for downstream processing

The key insight: when you combine screenshot + context (DOM tree, component hierarchy, or code state), you give AI agents enough information to make targeted fixes without guesswork.

The Workflow

Here's how Peekaboo plugs into an AI-assisted development workflow:

1. Capture Current State

# Take a screenshot of the running app
peekaboo image --app "YourApp" \
  --capture-focus background \
  --path ./screenshots/current-state.png

Peekaboo generates the screenshot file. You can optionally capture metadata by using the --json-output flag, which outputs structured information about the capture to stdout (useful for scripting)

2. Feed Screenshot to Agent

With the screenshot captured, you can now provide it as context to your AI agent:

I need help fixing the layout on the user profile page.
Here's a screenshot of the current state: [attach current-state.png]

The avatar should be centered in the header, but it's pushed to the left.
Can you identify what's causing this and suggest a fix?

The agent can now:

  • See the actual rendered output
  • Compare against the intended design
  • Identify specific CSS or layout issues
  • Propose targeted changes

3. Verify the Fix

After applying the agent's suggestions:

# Capture the updated state
peekaboo image --app "YourApp" \
  --capture-focus background \
  --path ./screenshots/after-fix.png

Now you have visual proof of the change. You can:

  • Compare before/after screenshots manually
  • Use image diff tools to highlight changes
  • Archive screenshots for regression testing
  • Share with team members for quick verification

Automation Patterns

Pre-Commit Screenshot Validation

Create a pre-commit hook that captures screenshots and checks for unexpected visual changes:

#!/bin/bash
# .git/hooks/pre-commit

# Capture current screenshots
./scripts/capture-screenshots.sh

# Compare with baseline
if ! ./scripts/compare-screenshots.sh; then
  echo "Visual regression detected!"
  echo "Review changes in ./screenshots/diff/"
  exit 1
fi

GitHub Actions Integration

Run Peekaboo in CI to catch visual regressions:

name: Visual Regression Tests

on: [pull_request]

jobs:
  visual-test:
    runs-on: macos-latest
    steps:
      - uses: actions/checkout@v3

      - name: Install Peekaboo
        run: brew install steipete/tap/peekaboo

      - name: Build and run app
        run: |
          # Start your app/simulator
          open -a Simulator
          # ... your build commands ...

      - name: Capture screenshots
        run: |
          peekaboo image --app "Simulator" \
            --path ./test-screenshots/pr-${{ github.event.pull_request.number }}.png

      - name: Upload screenshots
        uses: actions/upload-artifact@v3
        with:
          name: visual-snapshots
          path: ./test-screenshots/

Agent-Requested Screenshots

Create a script that agents can invoke to get fresh screenshots:

#!/bin/bash
# scripts/get-fresh-screenshot.sh

# Ensure app is running
./scripts/ensure-app-running.sh

# Capture screenshot
peekaboo image --app "$APP_NAME" \
  --capture-focus background \
  --path "./agent-screenshots/$(date +%Y%m%d-%H%M%S).png"

# Output path for agent to consume
echo "Screenshot saved: ./agent-screenshots/$(date +%Y%m%d-%H%M%S).png"

Scripting with JSON Output

Peekaboo's --json-output flag enables programmatic capture workflows:

# Capture and extract the file path
SCREENSHOT_PATH=$(peekaboo image --app "Safari" \
  --json-output 2>/dev/null | jq -r '.data.saved_files[0].path')

# Use in scripts
echo "Screenshot saved to: $SCREENSHOT_PATH"

The JSON output includes:

  • File paths of captured images
  • Window titles and IDs
  • Success/failure status
  • Debug information

This enables automated workflows where screenshots are captured and processed programmatically:

#!/bin/bash
# Capture screenshot and upload to issue tracking system
RESULT=$(peekaboo image --app "MyApp" --json-output)
SCREENSHOT_PATH=$(echo "$RESULT" | jq -r '.data.saved_files[0].path')

if [ -f "$SCREENSHOT_PATH" ]; then
  # Upload to bug tracker, send to AI agent, etc.
  curl -F "file=@$SCREENSHOT_PATH" https://api.example.com/upload
fi

Real-World Example: Layout Debugging

Let's walk through a real scenario where screenshots + AI saved hours of debugging.

Problem: After a React component refactor, users reported that the mobile menu was "acting weird" on iPad.

Traditional Debugging:

  1. Reproduce the issue (10 min)
  2. Inspect element in browser (5 min)
  3. Hunt through CSS files (20 min)
  4. Try various fixes (30 min)
  5. Test on different devices (15 min)

Total: ~80 minutes of developer time

Screenshot-Grounded Approach:

# 1. Capture the issue (30 seconds)
peekaboo image --app "Safari" \
  --path ./bug-reports/ipad-menu-issue.png

# 2. Ask agent for help (2 minutes)
# Upload screenshot + component code to AI assistant
# "The mobile menu renders incorrectly on iPad. See screenshot.
#  Component code attached. What's wrong?"

# 3. Agent identifies the issue immediately:
# "The menu uses `@media (max-width: 768px)` but iPad Pro
#  is 1024px wide. The breakpoint should be 1024px or use
#  a device-specific media query."

# 4. Apply fix (1 minute)
# 5. Verify with new screenshot (30 seconds)

Total: ~5 minutes

The screenshot provided the ground truth that allowed the agent to make a definitive diagnosis instead of suggesting generic debugging steps.

Best Practices

1. Keep Screenshots Fresh

Stale screenshots lead to wrong fixes. Implement a "screenshot expiry" system:

# scripts/check-screenshot-freshness.sh
SCREENSHOT_AGE=$(find ./screenshots -name "*.png" -mtime +1d | wc -l)

if [ $SCREENSHOT_AGE -gt 0 ]; then
  echo "Warning: Screenshots older than 24 hours detected"
  echo "Consider refreshing with: make screenshots"
fi

2. Tag Screenshots with Context

Name screenshots descriptively:

# Good
peekaboo image --app "MyApp" \
  --path "./screenshots/profile-page-mobile-portrait.png"

# Bad
peekaboo image --app "MyApp" \
  --path "./screenshots/screenshot1.png"

3. Version Control Screenshot Baselines

For key views, commit baseline screenshots:

screenshots/
  baselines/
    home-page-desktop.png
    home-page-mobile.png
    login-page-desktop.png
  current/
    # Generated on each test run
  diffs/
    # Visual differences from baseline

4. Combine with DOM Snapshots

Screenshots show what, DOM shows why:

# Capture screenshot
peekaboo image --app "MyApp" --path ./screen.png

# Capture DOM state
# (Using your preferred method - browser devtools, Playwright, etc.)

# Provide both to agent for maximum context

Limitations and Gotchas

Peekaboo is macOS-only: If you're on Linux/Windows, consider alternatives like:

  • Playwright's screenshot API
  • Puppeteer for web apps
  • Platform-specific tools (adb for Android, etc.)

Screenshot timing matters: Animations, loading states, and async content can cause non-deterministic screenshots. Use Peekaboo's wait/delay options or ensure app is in stable state.

File size: High-resolution screenshots can get large. Consider:

  • Compressing images with ImageMagick/pngquant
  • Limiting screenshot retention (keep last N days)
  • Storing only diffs for archival

Checklist: Setting Up Screenshot-Grounded Development

  • Install Peekaboo (brew install steipete/tap/peekaboo)
  • Create scripts directory for screenshot automation
  • Set up screenshot baseline directory structure
  • Configure CI/CD to capture screenshots on PRs
  • Establish screenshot naming convention
  • Create agent prompt templates that include screenshot context
  • Set up pre-commit hook for visual regression checks (optional)
  • Document screenshot workflow for your team
  • Test screenshot capture reliability across devices
  • Establish screenshot retention/archival policy

Claude Code Skill: Peekaboo

To make screenshot-grounded development even easier, I've created a Claude Code skill that packages the entire workflow into a reusable, portable format.

What's Included

The Peekaboo skill provides:

Scripts:

  • capture-screenshot.sh - Basic screenshot capture with validation
  • capture-with-json.sh - Screenshot capture with JSON metadata parsing

Templates:

  • screenshot-prompt.md - Guide for creating effective AI prompts with screenshots

Documentation:

  • Complete skill documentation (SKILL.md)
  • Usage examples and troubleshooting

Installation

# Clone the skill to your Claude skills directory
git clone https://github.com/rocksoup/blog.git
cp -r blog/skills/peekaboo ~/.claude/skills/

# Or download directly from:
# https://github.com/rocksoup/blog/tree/main/skills/peekaboo

How It Works

Once installed, Claude Code automatically loads the skill when appropriate:

User: "The header looks weird on iPad"

Claude: [Loads Peekaboo skill] "Let me capture a screenshot to see what's happening..."

~/.claude/skills/peekaboo/scripts/capture-screenshot.sh "Safari" ./screenshot.png

The skill includes contextual instructions that help Claude:

  • Know when to suggest screenshot capture
  • Use the right scripts for the task
  • Parse JSON metadata programmatically
  • Create effective screenshot-based prompts

Skill Benefits

  • Portable: Works across all your Claude Code projects
  • Automatic: Claude loads it when working on UI/frontend tasks
  • Consistent: Standardized workflow across projects
  • Extensible: Easy to add custom scripts and templates

Learn more about Claude Code skills at: https://www.anthropic.com/news/skills

Conclusion

Screenshots bridge the gap between what AI agents understand (code structure, patterns, conventions) and what humans see (actual rendered output). By making screenshots a first-class part of your development workflow, you transform vague "it looks wrong" feedback into precise, actionable guidance.

Peekaboo makes this practical: it's fast, reliable, and integrates seamlessly with existing tools. Whether you're debugging layout issues, catching visual regressions, or guiding AI assistants toward better UI implementations, screenshot-grounded development turns visual feedback into code.

The next time an AI agent suggests a UI change, don't just trust the code—ask for a screenshot. You might be surprised how much faster you ship when everyone can actually see what they're building.

Resources