> For the complete documentation index, see [llms.txt](https://docs.neurolabai.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.neurolabai.io/engineering-and-scientific-assurance/evaluation-methods.md). # Evaluation Methods Neurolab evaluates separate heads and routes with separate cohorts. There is no scientifically meaningful platform-wide accuracy number. ## Canonical evidence source Current serving evidence is bound to a versioned validation report. The report identifies the model version, knowledge-graph vintage, checkpoint hashes, metric inputs, generator state, and report checksum. Training documentation explains construction history; it does not override the served validation report's metrics. The known-head rows below use raw GNN output before GraphRAG, reranking, or served-route enrichment. They are measured on the historical pooled validation-plus-test **development** list. A later audit identified four train/test scaffold-family overlaps, so this cohort is not scaffold-disjoint and is not an untouched independent test. ## Known-compound raw-head results | Capability | Hit\@1 (95% CI) | Hit\@5 (95% CI) | n | Required interpretation | | ------------------- | ---------------------: | ---------------------: | --: | -------------------------------------------------------------------------------------------------------------------------- | | Cognition | 0.8520 (0.8010–0.9031) | 0.9745 (0.9541–0.9949) | 196 | Coarse cognition label ranking on a pooled development cohort. | | Reported usage | 0.5694 (0.4583–0.6944) | 0.7917 (0.6944–0.8889) | 72 | Ranking evidence, not efficacy, approval, or patient benefit. | | Side effects | 0.5517 (0.3793–0.7241) | 0.7931 (0.6207–0.9310) | 29 | Small cohort; top-1 loses to the provisional frequency control. Not a safety screen. | | ATC | 0.6897 (0.5172–0.8621) | 0.8966 (0.7931–1.0000) | 29 | Small cohort; checkpoint lacks bound ATC label metadata and serving uses a disclosed length-checked legacy reconstruction. | | Mechanism of action | 0.5941 (0.5314–0.6569) | 0.8117 (0.7615–0.8577) | 239 | Known-compound pooled development result. | The historical 95% intervals use 2,000 bootstrap resamples of compounds. They do not resample chemistry components or model seeds and should not be read as per-prediction confidence intervals. Current evaluation tooling supports chemistry-component cluster resampling for split-aware future studies; it does not rewrite these frozen historical intervals. ### What Hit\@K means `Hit@1` asks whether at least one gold label is the first-ranked output. `Hit@5` asks whether at least one gold label appears in the first five. These metrics do not measure effect size, probability calibration, causal mechanism, clinical usefulness, or completeness. ## Fine-cognition mechanism evaluation With curated/oracle receptor input, the CircuitFramework reasoner achieved Hit\@5 `0.875` with a 95% interval of `0.7500–0.9688` on `n=32`. This is a reused development cohort. It tests receptor-to-mechanism composition when the receptor input is already known; it is not an independent validation and does not evaluate structure-to-receptor recovery. The complete structure-to-receptor-to-reasoner cohort contains only three eligible cases, with one observed hit. The validation artifact therefore marks end-to-end fine cognition **UNVALIDATED** and does not permit an accuracy claim. ## Structure-to-receptor evidence The current served three-seed structure-to-receptor ranking metadata records top-1 `0.536` and Hit\@5 `0.727` on `n=284` from a Murcko-scaffold-OOD development sweep. These values are development-tuned build metadata, not a recomputed deployment-harness result. The same cohort was used to select a blend weight; compound identities and the evaluator are not packaged, and no confidence interval is available. Current densified profile AUROC and downstream cognition, MoA, or brain performance have not been remeasured. The older `0.882` AUROC claim belongs to a predecessor and is retired for the current served route. ## Brain evaluation boundary The current literature brain-map pilot contains one locked, network-compatible observation for memantine and the Default network. It is an oracle-receptor pipeline diagnostic across separate channels, not a structure-route evaluation. The review batch is incomplete and source-access-blocked. Intervals are not estimable, direction was not scored, and the validation report sets aggregate accuracy, directional score, and structure-route result to null. These results are insufficient to estimate brain-localization accuracy or justify a new release. ## Controls and baselines A rigorous evaluation should test more than headline rank: * exact random, training-frequency, 1-nearest-neighbor, and 5-nearest-neighbor controls; * label-support and fixed-K recall/F1; * correlated-seed agreement without treating seeds as independent experiments; * self-neighbor and near-analog exclusion; * chemistry-component, scaffold-, target-, drug-class-, and temporal-OOD splits; * oracle-receptor and end-to-end mechanism results reported separately; * population-average and other base-rate controls for brain maps; and * abstention precision and false-confidence cases. The fixed Donepezil/Fluoxetine/Morphine/Methylphenidate panel is a deployment regression check. It is not an independent accuracy estimate. ## Evidence needed for stronger claims Before stronger public claims, the project needs: * a frozen independent test after tuning is complete; * dual-disjoint structure cohorts that prevent both GNN and retrieval leakage; * temporal-prospective cases with pre-cutoff models and post-cutoff evidence; * independently recomputable structure-route metrics and cohort identities; * per-bucket calibration and abstention curves; * larger, source-complete brain-map evaluation; and * end-to-end validation from structure through receptor and mechanism outputs. ## How to cite a metric Every performance statement should include the route, head, metric, denominator, cohort/split, interval status, artifact version, and key limitation. For example: > On the v43.262 known-compound raw cognition head, Hit\@1 was 0.8520 (historical 95% compound-bootstrap CI 0.8010–0.9031; n=196) on a pooled development cohort. A later audit found four train/test scaffold-family overlaps, so this is not a scaffold-disjoint or independent-test result and is not a per-prediction probability. *** **Current evaluation boundary:** production-family metrics remain tied to v43.262 and to the cohorts, controls, intervals, and limitations reported above. Next: [Limitations and Known Gaps](broken://pages/5NlHxyDS31YmkP5FSGPY) --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.neurolabai.io/engineering-and-scientific-assurance/evaluation-methods.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.