AgentSkeptic: Verifying the Real Database State of AI Agent Workflows Using Read-Only SQL

Key Takeaways: AgentSkeptic is a tool that verifies the database state of AI agents and automated workflows using read-only SQL. It aims to solve the hidden 'silent failure' issue where tracking shows success but the database is not actually updated. The tool supports SQLite and PostgreSQL databases, offers two verification modes (contract verification and quick verification), and ensures the credibility of data operations by comparing the expected state with the actual database state.

In today's era of widespread AI agents and automated workflows, there is a hidden risk: workflow tracking logs and tool responses all show success, but database rows are missing, outdated, or incorrect (known as 'silent failure'). Causes include improper network timeout retry logic, incorrect partial failure handling, race conditions, and transaction rollbacks not being properly propagated. Traditional observability tools (logs, tracing, APM) can only tell whether steps have run, but cannot verify the expected values of database rows. This semantic verification gap is particularly dangerous in customer-facing or regulated scenarios.

AgentSkeptic's core design philosophy is: Tracking success does not equal real database updates; only row-level states verified via SQL queries are credible ground truth. Its technical architecture is based on a three-layer verification model:

1. Declaration Layer: Structured tool activities captured from workflows (in NDJSON format), including tool IDs, parameters, and other information;
2. Expectation Layer: Derives the expected state of the database based on registry rules or automatic inference;
3. Observation Layer: Obtains the actual database state via read-only SQL queries, supports SQLite and PostgreSQL, and has no risk of modification during verification.

AgentSkeptic offers two verification modes: Contract Mode (Recommended, Audit-Grade Reliable): Users need to provide a registry JSON file, defining verification rules (such as table names, identity matching conditions, required fields) for each tool ID. The engine converts these rules into SQL queries and compares the results; Quick Verification Mode (Zero Configuration, Exploratory Scenarios): Only requires tool activity logs and database connections. The system automatically infers rules, but results are not for audit purposes. Contract mode is preferred for strict scenarios.

Typical application scenarios of AgentSkeptic include:

• Release Blocking: Verify key data operations in CI/CD pipelines; block releases if verification fails;
• Manual Review Trigger: Automatically trigger manual reviews when inconsistencies are found;
• Incident Response: Quickly locate data inconsistencies and reduce troubleshooting time;
• Audit Tracking: Generate verification artifacts attached to audit logs to provide compliance evidence. In addition, the CI enforcement feature (agentskeptic enforce) requires actual verification results to match the lock file, ensuring the predictability of data operation behaviors.

AgentSkeptic's unique position in the observability tool spectrum:

Tool Type	Information Provided	Limitations
Logs/Tracing	Whether steps ran, duration, error messages	Does not guarantee database row state
Unit/Integration Tests	Correctness of code paths	Does not verify real database state in production
Metrics/APM	Health and latency	Does not verify semantic correctness of persisted records
AgentSkeptic	Whether observed SQL matches expectations	Does not prove actual tool execution or writing
It is suitable for scenarios requiring SQL ground truth verification, but not for proving tool execution, general log search, or non-SQL system verification.

AgentSkeptic's advanced features include:

• Cross-Run Comparison: Compare results from different workflow runs to identify abnormal patterns;
• Execution Tracing: End-to-end execution visibility to help understand complex workflow behaviors;
• In-Process Hooks: SQLite supports the withWorkflowVerification function for in-process integration verification;
• Run Packages and Signatures: Package workflow records and encrypt them with signatures to ensure tamper-proof audit tracking;
• Debug Console: Interactive debugging interface to assist in developing and troubleshooting verification rules;
• Guarantee Subsystem: Versioned manifest multi-scenario scanning, supporting timestamp expiration checks.

AgentSkeptic uses an open-source core + commercial extension model:

• Open Source Version (GitHub): Provides full verify functionality, no API key required, suitable for local development, forking, and offline use;
• Commercial Version (npm package agentskeptic): Adds features like batch processing, quick verification, CI lock flags, and the enforce command on top of OSS. It requires a subscription and API key. The layered strategy ensures core capabilities are open while meeting advanced enterprise needs.