Building CyberCortex AI and Testing Its Security Research Capabilities

When I started building CyberCortex, I wanted to explore a practical question: how can an AI assistant participate in cybersecurity research while keeping its actions controlled and its conclusions tied to evidence?

Producing a convincing explanation is only one part of that problem. A security research system also needs to discover an application's exposed functionality, recognize authentication boundaries, choose appropriate tests, stay within its authorization, and preserve enough evidence for someone else to review the result.

CyberCortex has developed around those requirements. The work now includes a modular agent platform, controlled verification workflows, a local testing range, persistent research records, and blind benchmarks. GraphQL testing has been particularly useful because it exposes weaknesses in the connections between discovery, evidence, ownership, and execution.

This is a development account of the system so far. Some capabilities have passed controlled tests; others still need stronger live validation.

What CyberCortex is

CyberCortex is an AI powered cybersecurity agent platform. Its original local configuration combines Python, Ollama, DeepSeek R1 Distill 32B, and custom security tools. The project has also used FastAPI as part of its technology stack.

I built the application architecture, workflows, integrations, policy controls, and evidence handling around an existing language model. That distinction matters when explaining what I mean by “my AI”: the CyberCortex engineering work concerns the system around the model, rather than a claim that I pretrained DeepSeek's model weights.

The initial platform supports reconnaissance, HTTP analysis, crawling, parameter analysis, misconfiguration observations, JavaScript analysis, authorization test planning, and report generation. Katana and Nuclei appear in the original tool integrations. Their availability depends on the execution mode: later policy work deliberately blocked legacy network and subprocess paths that could not preserve the shared authorization and request accounting boundary.

A capability appearing in a tool registry therefore does not mean it can run automatically in every assessment.

How I organized the files

The architecture separates entry points, planning, tools, policy, evidence, and reporting. This makes it possible to improve a component without treating the entire agent as one large script.

Several later modules illustrate how the design became more explicit. agent_core/verification_capabilities.py declares capability maturity and execution requirements. agent_core/verification_runtime.py provides a shared verification runtime. agent_core/request_budget.py defines request accounting contracts. agent_core/result_normalizer.py handles public result serialization and sanitization.

The GraphQL work includes graphql_discovery.py, graphql_ingest.py, and graphql_readiness.py within the research package. These separate finding a service, importing its semantics, and deciding whether the available evidence is sufficient for further research.

The point of these boundaries is practical: a discovered endpoint, a proposed test, an authorized experiment, and a confirmed finding are different objects with different requirements.

How an assessment develops

The initial workflow validates scope, creates a plan, collects baseline observations, analyzes the discovered surface, and generates a report. Later research orchestration extends this into a lifecycle involving hypotheses, candidate selection, experiment compilation, authorization, execution, evaluation, and possible reproduction.

For example, an ID-like parameter can support an object authorization hypothesis. It does not establish a vulnerability. The system also needs a legitimate object reference, a controlled owner, an appropriate comparison identity, and an authorized request that can test the hypothesis.

If those requirements are missing, the correct result may be an explanation of the missing evidence. A speculative finding would make the report less useful.

Guardrails implemented in code

The security boundary needs to operate even when model output is malformed or overconfident. CyberCortex therefore uses deterministic checks around the model and the tools.

Scope and transport authorization

The Phase 2 hardening work moved policy and request accounting ahead of discovery. A selected assessment policy, a shared request budget, and a scoped HTTP client govern the authorized network path.

Checks include the target URL, scheme, port, allowed method, exclusions, controlled accounts, resource ownership, concurrency, and rate limits. Redirect destinations need authorization before another request is attempted. Legacy routes that could bypass the shared boundary were disabled or made available only for offline planning.

One lesson was that protecting the final verifier is insufficient if discovery can already send traffic outside the selected policy.

Strict inputs and controlled identities

Verification inputs use category-specific strict models. Unknown fields, coercible booleans, malformed selectors, and inconsistent bindings must be rejected before traffic is sent.

Object authorization comparisons use controlled accounts and an owner-bound test object. A denied comparison request only supports a secure conclusion when a valid owner baseline establishes that the object and protected evidence were accessible under the expected identity.

Likewise, a 200 OK response alone does not prove an authentication bypass. The response needs the relevant protected evidence and the correct identity and resource relationships.

Request budgets and cleanup

The request ledger accounts for discovery, authentication, verification, and cleanup. Later result contracts calculate a per-result delta from authoritative before-and-after snapshots, keeping an individual experiment's cost separate from the whole assessment.

An authorized transport attempt that fails still consumes budget. A request blocked before dispatch has zero transport cost. Cleanup traffic also counts.

Stateful verification needs a restoration strategy. The controlled mass-assignment workflow, for example, includes restoration and an independent confirmation request. Recovery workflows retain explicit intermediate states rather than pretending that pending cleanup is a finished verification.

Secrets and public evidence

Credentials belong in controlled runtime handling. Public results, reports, logs, and model evidence packages pass through a recursive sanitization boundary.

The hardening tests use sentinel secrets to check whether sensitive values escape through nested objects, diagnostic strings, URLs, exceptions, exports, or reports. Useful telemetry, such as counts and safe cookie attributes, must remain available without retaining credential values.

This also creates a design challenge: sanitization must remove secrets without destroying the evidence relationships needed to understand a test.

The five development phases

The five-phase roadmap gives the project a direction, while the implementation records describe what was actually available at each checkpoint. The phases below reflect that roadmap; they are not a claim that all five are finished.

The phases also changed how I evaluated progress. Early work asked whether a tool could collect useful information. Later work asks whether an entire investigation preserves policy, evidence, accounting, and reproducibility across multiple components.

Building a local vulnerable testing range

I built a separate environment called cybercortex-range to test the agent against deliberately vulnerable application behavior. The range provides a local place to investigate web and API security categories associated with OWASP-style testing.

The preserved configuration contains separate API, authentication, GraphQL, JWT, injection, upload, business logic, and server-side labs. It also includes an internal service and object storage infrastructure. A full claim of coverage for every category of a particular OWASP Top 10 edition needs an explicit mapping to the range's fixtures.

Docker Compose builds the Python lab applications using infrastructure/python-lab.Dockerfile and application directories under apps/. The configuration binds published lab services to 127.0.0.1.

The documented containers also use resource limits, dropped Linux capabilities, read-only root filesystems, temporary storage, health checks, and no-new-privileges. Selected applications use data volumes for their state. The internal service is exposed within the container network without a published host port in the inspected configuration.

This gives the testing environment its own boundaries even though the applications intentionally contain weaknesses.

The range and the later blind GraphQL benchmark fixture are related testing efforts, but they are distinct environments. The blind benchmark procedures launch a dedicated FastAPI/Uvicorn GraphQL fixture on 127.0.0.1:8765.

How I tested the implementation

Testing has covered individual contracts, shared runtime behavior, persistence, and complete benchmark attempts. The important question is what each test establishes.

The quality gates recorded during development include Black, Ruff, Python compilation, pytest, and Git whitespace checks. Test counts grew as the architecture developed. The Phase 2 completion checkpoint reported 849 passing tests, and the later P4-1I.34 checkpoint reported 2,487.

Those are dated regression checkpoints. A passing suite does not establish that every function has exhaustive coverage or that every supported vulnerability category works against a live target. For that, the actual executed path and its resulting evidence matter.

An August 20 baseline assessment makes the distinction clear. It reported 92.9% workflow coverage, six observations, and zero verified findings. Nuclei timed out, and the generated model analysis failed grounding validation, so the system wrote a deterministic fallback report. The assessment ended as completed with limitations.

The coverage figure described that workflow's execution coverage. It was not a vulnerability detection accuracy score.

The scoring systems I used

CyberCortex has used different measurements for different questions. Keeping them separate avoids turning one percentage into an unsupported claim about the whole system.

The architecture completion audit

The original Phase 2 independent completion audit awarded 68 out of 100 points across nine dimensions.

The total is the sum of the awarded points. It is a weighted engineering audit, rather than a measured probability of finding a vulnerability.

The audit identified three P0 and six P1 tasks required before freezing Phase 2. Subsequent hardening addressed those areas. The original 68-point score remained a historical audit result; it was not automatically recalculated after each fix.

The range benchmark scorer

The range scorer in benchmarks/score.py compares normalized findings with a ground-truth vulnerability inventory. Matching uses application, canonical category, and affected functionality. A finding's self-declared ground-truth ID is not trusted as the match.

It calculates:

The implementation returns zero when a metric's denominator is zero. It deduplicates matched vulnerabilities and prefers a verified record over an earlier unverified match. Verification credit also requires the scorer's evidence requirements to be satisfied.

For illustration only, ten ground-truth vulnerabilities, six matched discoveries, two unmatched findings, and four sufficiently evidenced verified matches would produce 75% precision, 60% recall, and approximately 66.7% verification rate. These numbers are an explanation of the formulas, not a CyberCortex run result.

The inspected scorer also accepts optional reported metrics and can derive time to first finding from finding timestamps. Its formulas describe that version of the range scorer; they should not be silently substituted for a later GraphQL benchmark policy.

Why GraphQL became a major benchmark

GraphQL forces the system to distinguish service discovery from application understanding. Finding a GraphQL route establishes that a service may be reachable. Introspection can expose types, fields, and arguments. Neither establishes that an authorization vulnerability exists.

An application operation needs its own provenance. CyberCortex's research design requires observed or explicitly trusted typed operation evidence before building executable operation templates. Discovery probes such as __typename remain discovery evidence.

A schema can describe an object field without establishing which observed application request uses it, which variable identifies a controlled object, or which identity owns that object. Those relationships are essential for a meaningful authorization experiment.

Keeping the benchmark blind

The blind procedures prohibit manually supplying the GraphQL endpoint, schema, application operation, field, object ID, vulnerability hypothesis, or expected vulnerable behavior to the research process. Controlled accounts provide authorized context, while hidden ground truth belongs to the evaluation side.

The system must obtain the application evidence through legitimate discovery. If the lab exposes useful operation material but the agent cannot acquire and register it, that is a measurable integration gap.

What the preserved runs showed

Run 2 reached a generic REST authentication experiment, which returned an inconclusive result. It then stopped because no eligible experiments remained. Its research ledger recorded zero model calls, so the attempt cannot be described as a successful DeepSeek reasoning benchmark.

Run 6 accounted for 17 attempted requests: 11 ordinary discovery requests, two GraphQL discovery requests, one public asset discovery request, one public asset retrieval request, and two controlled-object acquisition requests. None were verification, reproduction, cleanup, or state-changing requests.

The failure happened after a public client asset was retrieved but before an operation was durably registered. That is a different failure from being unable to reach the server, and a different result from testing an operation and finding it secure.

Preserving these distinctions makes the benchmarks useful. They show where the investigation stopped and what evidence was missing at that point.

Strengthening the GraphQL evidence boundary

The later P4-1I.61 checkpoint addressed authoritative GraphQL evidence trust. The reported controls bind trusted producer evidence to request identity, HTTP method, endpoint, final URL, status, content digest, and acquisition context. Modified or serialized copies lose their authority.

The checkpoint also reports rejection of self-asserted confirmations, request-only ownership claims, forged records, conflicting object-reference reuse, and incomplete variable-binding evidence. REST-to-GraphQL correlation remains possible when the supporting evidence satisfies the required relationships.

These controls address a central concern in an AI research system: data that looks plausible must not grant itself authority to drive a security experiment.

The checkpoint reports a completed positive synthetic lifecycle. That supports the tested evidence-handling paths. A live blind run reaching experiment execution, reproduction, and finding confirmation remains a separate demonstration that must be reported from its own artifacts.

What I have learned so far

The most demanding work has been connecting components without losing the meaning of their evidence. A response can arrive successfully while its object binding is unusable. A schema can be available while no application operation is ready. An assessment can finish while finding no verified vulnerability.

CyberCortex's development has made those distinctions explicit in its architecture, tests, and benchmarks. Progress comes from making the system's behavior easier to inspect: what it observed, why it proposed a test, what authorized the request, how much it cost, and what the result actually proved.

The next useful milestone is a preserved live benchmark that demonstrates the complete intended research path under the same controls. That is the evidence needed to extend the claims beyond the components and controlled scenarios already tested.