Embedding | kBench

Quality

Recall@10 (%): The share of judged relevant documents found in the first 10 results, averaged over queries and repeats. Any positive relevance grade counts. Higher is better.
MRR@10 (%): Mean reciprocal rank of the first relevant result within the top 10. No relevant result gives zero. Scores are shown as percentages. Higher is better.
Precision@1 (%): The share of queries whose first result is judged relevant, averaged over repeats. Any positive relevance grade counts. Higher is better.
Accuracy (%): The quality score of the dataset. Each benchmark uses one named score. Do not compare different score names. Higher is better.
nDCG@10 (%): Normalized discounted cumulative gain at 10. It scores the order of the first 10 search results against graded relevance. 100 is a perfect order. Higher is better.
Hit@1 (%): The percentage of queries where the first result is a known target. The known targets are not a complete list of relevant results. Higher is better.
Decision accuracy (%): The percentage of requests where the most probable answer is the correct answer. For yes or no questions, the answer is yes when its probability is 0.5 or more. A request without an answer counts as wrong. Higher is better.
Brier score: Calibration error: the squared difference between the given probabilities and the correct answer, averaged over answered requests. 0 is perfect. For multiple choice, the squares of all options are added, so the maximum is 2. Always guessing evenly between three options gives about 0.67. Lower is better.
Character accuracy (%): 100 minus the character error rate, with a minimum of 0. Higher is better.
CER (%): Character error rate. The character edits that make the output equal to the reference text, divided by the reference length. Insertions can make it more than 100%. Lower is better.
WER (%): Word error rate. The word edits that make the output equal to the reference text, divided by the number of reference words. Lower is better.

Testing

Token limit: The token budget for an input or the model context. The evidence of each run gives the exact rule. Embedding tests cut longer text.
Batch size: The number of inputs that the model gets in one call.
Not measured: The test did not record this value. It is not zero.
Evidence: The saved source file for a result. Open it to see the raw measurement.
Mean, median and p95: The mean is the average. The median is the middle value. p95 is the value that 95% of the measurements do not exceed.
Context: The tokens that a model can hold for one request. For OCR, the context must also hold image tokens and the generated text.
Warmup and repeat: A warmup runs the model before timing starts. A repeat runs the same test again to collect another measurement.

Models

Embedding model: A model that changes text into a vector. Search compares vectors to find related text.
Vector: A list of numbers that a model makes for one text. One input gives one vector.
Decision model: A model that answers a typed question about a text and gives a probability for each possible answer. It does not write text. Question types are yes or no, multiple choice and score.
OCR: Optical character recognition. An OCR model reads the text in an image of a page.
Token: A small part of text that a model reads, often a part of a word.
Number format: The number format of the model weights. FP32 uses 32-bit numbers. FP16 and BF16 use 16 bits. Q8 and Q4 keep weights in 8 or 4 bits. Fewer bits use less memory and can change quality. This is not review precision.
Vector size (dimensions): The number of values in one vector. A larger vector uses more storage. Vector size is not a quality score. This is not a ranking.

Hardware

CPU: Central processing unit. The main processor of the computer.
GPU: Graphics processing unit. Here, a separate graphics card with its own memory.
iGPU: Integrated GPU. The graphics part inside the processor. It shares the main memory.
Engine and threads: The engine is the hardware and software that run the model. A thread is one path of work in the processor. More threads can change speed and memory use.

Speed

Tokens per second (tokens/s): Input tokens that the model reads in one second. Embedding tests count text tokens without padding. OCR tests can include image tokens. The evidence of each run has the counting rule. Higher is better.
Vectors per second (vectors/s): Vectors that the model makes in one second. Higher is better.
Pages per second (pages/s): Pages that the OCR model reads in one second. Higher is better.
Latency (ms): The time for one measured call, in milliseconds. The evidence of each run names the call and the statistic. Lower is better.

Memory

Peak RAM (MiB): RAM means random access memory: the main memory of the computer. This is its highest use during the test, in mebibytes (1 MiB is 1,048,576 bytes). Some results add client and server memory. Lower is better.
Peak VRAM (MiB): VRAM means video RAM: memory used by the graphics card. This is its highest use during the test, in mebibytes. An empty value is unknown, not zero. Lower is better.

UI

UI-Bench: UI means user interface: the visible controls and output of an app. UI-Bench tests how each model builds one landing page and five page designs from the same prompt.
Capture: A saved screenshot of a generated page. Desktop captures are 1440 by 900 pixels. Mobile captures are 390 by 844 pixels.
Not scored: UI-Bench has no scoring rule yet. No score exists. The criteria are requirements from the prompt.
Run metrics: Sessions, wall time, API time, tokens and cost of the agent run. UI-Bench reads them from local agent logs. They are not imported.
Development script: A Vite "dev" script in the project. UI-Bench needs it to start a preview. This is a static file check, not a run result.

Code review

Target hits (of 27): Found target issues out of 27 chances: 9 known issues in 12 public pull requests, each run 3 times. A hit counts when a finding title matches the target pattern. This is not precision or recall. Higher is better.
After wording check (of 27): Target hits after a read of the finding text. The read found targets that the title pattern missed. A coding agent did this check, not an independent person. Higher is better.
Known bugs found (of 42): Known reference bugs that a review found, out of the 42 that both judges accept as real, single bugs within the review scope. The solid count needs both judges to agree; the faded extension counts bugs only one judge credits. Other real findings are not counted here. Higher is better.
Valid findings: Findings that both judges rate as a real defect, described correctly, within the review scope. This includes real bugs that are not on the known list. Higher is better.
Disputed findings: Findings where the judges disagree, or at least one judge is uncertain. They count as neither valid nor invalid. This is not a ranking.
Invalid findings: Findings that both judges reject, or both mark out of scope: a wrong claim, a wrong mechanism or no concrete defect. Lower is better.
Median time per review (s): The median wall time of the original review call. Claude times include the Claude Code client; OpenAI times are one Responses API request. Judge time is not included. Lower is better.
Model-judged: GPT-6.1 Sol and Claude Opus 5.5, both at high effort, independently judged every saved finding. The verdicts are frozen and the scores are recomputed offline. No person checked them, and a judge can favor its own model family.
Reasoning effort: A model setting that controls how much the model thinks before it answers: none, low, medium, high or xhigh.
Seconds per review (s): The mean time for one review. Times from different dates are only indicative. Lower is better.
Cost per review (USD): The estimated cost of one review in US dollars at standard API prices. Lower is better.
Findings: All issues that the model reported. More findings is not better. This is not a ranking.
Pull request: A proposed code change for review.
Precision (review): The share of reported findings that are real issues. kReview keeps it empty until a person checks every finding. Not measured. Higher is better.
Recall: The share of known real issues that the model found. Not measured. Higher is better.
F1: One score that balances precision and recall. Not measured. Higher is better.
Severity: How serious a finding is: critical, high, medium, low or suggestion. Severity recall is the recall for critical and high findings only. Not measured.
Verdict: The decision of a person about a finding: accepted, dismissed, wontfix or unresolved.
Failure category: The step where a missed issue got lost, for example retrieval, generation or the verifier.
Readiness minimums: The minimum checked data that kReview needs before it compares finding quality.

Files and services

API: Application programming interface. The way one program sends a request to another program. API time measures time spent on these requests. As an engine, API means a hosted model service. The provider hardware is unknown, and the time includes the network from the client host.
CSV: Comma-separated values. A text file with rows and columns that a spreadsheet can open.
JSON: JavaScript Object Notation. A text file that stores named values, lists and records for a program to read.
PNG and JPEG: Image file formats. PNG keeps every pixel value. JPEG makes smaller files but can change fine image detail. UI captures use JPEG copies; source PNG hashes remain in the catalogue.
SHA-256 hash: A file check value made with Secure Hash Algorithm 256. The same bytes give the same value. Use it to check that a file did not change.