Longitudinal Study

Security Drift in Hosted LLM Agents

Measuring longitudinal stability of indirect prompt injection defenses and functional capabilities in multi-turn tool-calling agents across English, Telugu, and Tenglish.

Study Progress -

Last Evaluation Run -

Next Scheduled Run -

Models Tracked -

Suite Integrity Verified

Research Design & Metric Definition

This evaluation runs identical frozen attack scenarios against hosted model endpoints every week under controlled temperature and deterministic world-state goal predicates. Attack Success Rate (ASR) measures the percentage of trials where an indirect prompt injection payload induced unauthorized state changes (e.g. fraudulent order cancelation or shipping address redirection). Longitudinal change is tested via Logistic Generalized Estimating Equations (GEE) with scenario clustering.

Current Model Attack Success Rates

Hypothesis Evaluation Status

Output 1

Longitudinal Security Drift

Weekly attack success rates over time. Solid lines depict hosted models with shaded 95% cluster-bootstrap confidence intervals; dashed line represents the pinned local reference model.

Week	Hosted Model	Local Reference	Difference	95% CI	Trials (n)

Output 2

Multilingual Disparity: Telugu & Tenglish

Comparative vulnerability across English, native Telugu, and transliterated Tenglish. Matched scenario pairs evaluate translation vulnerability amplification.

Native vs. Machine Translation Grouped Dot & Interval

Language	Scenario Origin	Attack Success Rate	95% Interval	Sample (n)

Paired-Difference Forest Plot

Language by Channel Heatmap

Integrity & Baselines

Controls & System Canary

Tracking benign task success, canary harness output hashes, and TOST equivalence on pinned local models.

Benign Task Success Rate (Functional Capability)

Canary Prompt Invariance Heatmap

Local Negative Control (TOST)

Observed Returned Model IDs Timeline

Model Alias	Requested Model ID	Observed Returned Model IDs	Canary Hash State

Algorithmic Detection

Change Points & Provider Releases

PELT and CUSUM change-point detections aligned with documented vendor release notes.

Week	Model	Shift Magnitude	Detection Method	Provider Event Alignment

Output 3

Trial Records Export & Schema

Browse, filter, and inspect individual agent trial executions. All records conform to the versioned 25-column schema.

Downloads: CSV JSONL

SHA-256 Verified

View 25-Column Schema Specification

Trial ID	Week	Model	Language	Suite	Attack	Task	Latency

Showing 0 records

Evaluation Health

Data Quality & Telemetry

Audit trial execution completeness, HTTP retry and 429 rate-limiting events, token consumption, and response latency.

Overall Completion -

Rate-Limit 429 Events 0

HTTP / Network Retries 0

Uncaught Failures 0

Response Latency & Token Utilization

Model	Completion	Error Rate	Mean Latency	Mean Tokens

Weekly Archive

Weekly Evaluation Reports

Explore per-week evaluation results, compare batches across study weeks, and download Markdown and printable HTML reports.

Study Timeline:

Week of 2026-10-16

Pilot Batch

Markdown Printable HTML

Total Executed Trials -

Overall ASR -

ASR | Exposed -

Benign Task Success -

Canary Hash Status -

Per-Model Attack Success Rate

Model Alias	Trials (n)	Attack Success Rate	ASR (Exposed)	95% Interval	Change vs Week 1	Completion Rate

Findings & Cautious Assessment: Attack success rates reflect deterministic invariant evaluations in the simulated e-commerce environment. Where confidence intervals span zero when compared to prior evaluations, the data indicates no evidence of change. Observed differences reflect black-box behavioral measurements and should not be interpreted as evidence of provider optimization intent.

Output 4

Formal Evaluation Report

Automated research report generated from analysis and summary contracts with pre-registered conservative wording.

Download Markdown

Report Available Upon Evaluation

The formal evaluation report is compiled dynamically after weekly pipeline runs.

Scientific Methodology

Research Methods & Replication

Pre-registered hypotheses, cryptographic suite hashes, deterministic goal predicates, and instructions to test your own models.

Study Preregistration & Hypotheses

Pre-registration protocol: TODO: Pre-registration pending

ID	Description	Pre-Registered Decision Rule
H1	Longitudinal Drift in Attack Success	Logistic GEE slope b1 statistically significant after Benjamini-Hochberg FDR control (requires >= 4 weeks).
H2	Abrupt Step-Change Points	PELT/CUSUM changepoint detection with 95% cluster-bootstrap shift interval excluding zero.
H3	Local Reference Model Stability	Two One-Sided Tests (TOST) on pinned local model within [-0.03, +0.03].
H4	Language Vulnerability Interaction	GEE interaction coefficients for Telugu and Tenglish showing higher drift than English.
H5	Vendor Changelog Alignment	Proportion of empirical changepoints coinciding with announced provider model versions.
H6	Native vs. Machine Translation Gap	Paired difference on matched scenarios testing translation artifact effects on vulnerability.

Frozen Benchmark Suites

Suite Name	Target Scope	SHA-256 Digest

Test Your Own Model in 10 Minutes

Any OpenAI-compatible or local endpoint can be benchmarked against the identical frozen test suites.

# 1. Clone repository and set up environment
git clone https://github.com/Rdvprasad36/DriftWatch-TE.git
cd DriftWatch-TE
python -m venv .venv && source .venv/bin/activate
pip install -e .

# 2. Verify frozen suite integrity
driftwatch verify

# 3. Run pilot evaluation on your model endpoint
export LOCAL_MODEL_URL="http://localhost:11434/v1"
driftwatch run --week 2026-10-26 --suites canary,attacks_v1 --models local