A connected research program · Shiyu Ni · 2024–2026
Knowledge boundaries. Honest AI.
A capable model should also recognize the limits of its answers.
We study how language models assess their knowledge, why they become overconfident, and how to learn confidence that remains reliable across tasks and capability changes.
How can models recognize what they know—and what they do not?
Knowledge-boundary perception is the ability to recognize one’s knowledge limits. We study it through answer correctness under specified tasks and generation settings. Two complementary criteria matter: calibration—whether confidence matches empirical correctness rates—and discrimination—whether likely errors can be separated from correct answers. Our honesty-alignment work develops more reliable confidence estimates.
Reliable AI requires both task competence and accurate awareness of its limits.
These estimates can inform decisions. Adaptive retrieval is a tested application in our work; abstention, model routing, and reasoning-budget allocation are broader uses. Claim-level feedback for self-improvement motivates our longer-term agenda.
The regions and plots are schematic, not a map of stored knowledge or empirical results. Click to read at full size.Open original ↗
02 / Developing the research question
Evaluating, explaining, and improving knowledge-boundary perception.
We ask whether a model can distinguish its correct answers from its errors, and whether confidence reflects the likelihood of correctness. The program connects evaluation, signal analysis, explanations of overconfidence, and confidence learning across tasks and capability-training settings.
01 / Understanding knowledge boundaries
Can models recognize when their own knowledge is insufficient?
We first ask how accurately a model recognizes its own knowledge limits. We evaluate the mismatch between answer correctness and confidence, then investigate which verbal, probabilistic, and internal signals better reveal likely errors.
Answering correctly and recognizing correctness are distinct abilities.
Can a model distinguish questions it answers correctly from those it answers incorrectly? We separate QA accuracy from binary confidence–correctness agreement, measuring confident errors and correct answers marked uncertain. We find substantial overconfidence; prompting that combines caution with answer explanations improves this measured agreement.
As an application, we trigger retrieval when the prompted model expresses uncertainty. This combined approach can preserve or improve factual QA performance with fewer retrieval calls in the reported settings. The result reflects changes in both answer quality and confidence, and depends on the quality of retrieved evidence.
Enlarge figure ↗The paper measures confident errors separately from QA accuracy and compares them across prompting interventions. Its binary agreement measures are distinct from probabilistic calibration.
Supporting evidence · Correct answers vs. expressed certainty
Natural Questions · vanilla prompting · models evaluated in the 2024 study. Share of answers (%).
‘Certain’ is a binary self-report, not an average probability. Gap = certain rate − accuracy (percentage points).
This aggregate gap is not the paper’s sample-level Overconfidence metric. All five Table 2 models are shown.
Knowing vs. thinking you know
Setting
Actually correct
Says ‘certain’
Vicuna
26.34
97.22
LLaMA2
29.86
83.16
GPT-Instruct
40.03
81.0
ChatGPT
38.5
70.83
GPT-4
48.96
81.06
↓Expressed confidence reveals a gap in self-assessment. Is more useful information about that gap already present in the model’s probabilities and internal states?
Question-only states support correctness prediction before answer generation.
Alongside our comparison of confidence expressions, we investigate what supervised correctness predictors can extract from internal representations. Predictors trained on question-only states already identify likely success or failure; generated-answer states can provide additional information. These are learned estimates of correctness, not direct access to a model’s knowledge.
Confidence Consistency-based Calibration (C³) uses consistency across multiple-choice reformulations to revise post-generation confidence predictions. It improves error detection without revising the original answer. The work distinguishes two opportunities: earlier correctness assessment and more reliable post-generation assessment.
Enlarge figure ↗Supervised correctness predictors read representations available before or after answer generation. Predictive information is already present in question-only states.
↓Identifying predictive signals raises a related question: what patterns explain confidence errors, and how can those errors be reduced?
02 / Explaining and mitigating overconfidence
Why are models confidently wrong?
Predictive information alone does not ensure reliable confidence. We investigate popularity-related patterns in confident errors and use those patterns to inform targeted calibration.
Popular wrong answers can still attract high confidence.
Why are some wrong answers assigned high confidence? We distinguish the popularity of ground-truth answers from that of the model’s generated answers in factual QA. Generated-answer popularity and its co-occurrence with the question are associated with confidence even when the answer is wrong.
Incorporating these popularity signals improves confidence estimation across the evaluated models and datasets. The contribution is an explanation that informs a remedy: examining the answer the model actually generates helps identify systematic confidence errors and mitigate them.
Enlarge figure ↗A popular generated answer can be wrong. Separating its popularity from the popularity of the correct answer helps explain confidence errors.
↓A complementary direction concerns the learning process itself: how can reliable confidence be learned with fewer correctness labels, or from capability-training experience?
03 / Learning paradigms for reliable confidence
Two complementary paradigms for confidence learning
EliCal separates confidence elicitation from correctness-based calibration, reducing annotation requirements while supporting transfer across tasks. CoCal learns a separate confidence companion from capability-training experience, sharing that experience with task learning while keeping their parameters and objectives distinct.
Separating elicitation from calibration makes correctness labels go further.
Correctness labels are costly when confidence must transfer across tasks. EliCal trains a pre-generation confidence predictor from internal representations while keeping the answer-generating model fixed. It first learns from large-scale self-consistency targets, then calibrates with a small correctness-labeled set.
On HonestyBench, spanning ten free-form QA datasets, EliCal approaches fully supervised alignment with 1,000 correctness annotations; its confidence estimates also transfer to unseen MMLU tasks. This addresses annotation cost and cross-task transfer, complementing the study of specific confidence biases.
Enlarge figure ↗EliCal separates low-cost confidence elicitation from correctness-based calibration, with HonestyBench as the supporting data resource.
↓The second paradigm changes where confidence supervision comes from: the verified experience generated during capability training.
Paradigm 02 · Shared experience, separate learning
Capability-training experience can also train a separate confidence companion.
Reinforcement learning with verifiable rewards (RLVR) produces answers together with correctness feedback throughout capability training. CoCal uses the rollout hidden states and verified outcomes to train a lightweight confidence companion. Task and confidence learning share experience while retaining separate parameters and objectives.
Experiments on Qwen3-8B and Qwen3-14B show improved companion confidence estimates while preserving task performance; CoCal also outperforms matched post-hoc calibration in the Qwen3-8B comparison. This studies a different source and timing of confidence supervision: the trajectory of capability training, rather than only its final model.
Enlarge figure ↗CoCal reuses capability-training experience without coupling confidence learning to policy optimization. Figure 1 from the CoCal preprint.
Enlarge figure ↗The same RLVR rollouts support task-policy optimization and direct supervision of a separate confidence companion. Figure 2 from the CoCal preprint.
Scope. The companion reads hidden states and estimates confidence separately from the model’s verbal response. Experiments replay cached RLVR experience chronologically to study this learning setting.
What connects these studies
The common target is confidence that is informative about correctness and calibrated to it. Evaluation makes failures visible; signal analysis and explanation characterize them; two complementary learning paradigms address supervision cost and capability-training experience. Applications test whether better estimates improve decisions. Together, these contributions connect the evaluation of knowledge boundaries to reliable confidence and evidence-informed decisions.
03 / Broadening the scope
Broader settings and evaluator reliability
We extend self-assessment to visual inputs, multiple valid answers, and ranked outputs. A related line studies external factuality judges—the evaluators on which assessment itself may rely.
Does visual input change how a model recognizes its limits?
Across three vision-language models and three VQA datasets, verbal confidence is often overconfident, while token probabilities and answer consistency provide more reliable signals. Calibration methods improve perception, extending the evaluation agenda to knowledge grounded in both images and text.
What does disagreement mean when several answers are correct?
Our MACE benchmark exposes a failure of common confidence estimates: valid answer diversity can lower estimated confidence even as accuracy rises. Semantic Confidence Aggregation combines confidence over sampled responses to improve calibration, broadening the notion of reliable self-assessment beyond single-answer questions.
How do reasoning chains affect an external judge’s assessment of answer factuality?
Providing reasoning chains can help stronger judges, but fluent reasoning can also make incorrect answers look credible—even to strong judges. This examines the reliability of model-based factuality evaluation, an important counterpart to the answering model’s own self-assessment.
Can a reranker assess the quality of the ranking it just produced?
Self-consistency across sampled rankings provides competitive quality estimates, while direct verbal confidence is strongly overconfident. Supervised methods produce calibrated estimates with a few extra output tokens, carrying self-assessment from answer correctness to ranking quality.
A broader view of research agents. In Deep Research: A Systematic Survey, I contributed the sections on retrieval timing and its open challenges (§3.2.2 and §6.1), connecting knowledge-boundary perception to reliable information-seeking decisions.
04 / Research outlook
From “I may be wrong” to “here is what would help.”
From judging correctness to diagnosing what would make an answer correct.
An open-ended answer can mix correct, uncertain, and incorrect claims. Our next goal is to locate those differences, identify the missing knowledge or reasoning, and specify what evidence or action could resolve them.
This would turn self-assessment into useful feedback for retrieval, clarification, and revision. Independently verified corrections could then provide learning signals for further improvement. Connecting diagnosis, correction, and learning is the next research question.
LocateWhich claim is uncertain or wrong?
DiagnoseWhat knowledge, evidence, or reasoning is missing?
CorrectWhat action would resolve the gap?
Verify & learnDid the correction work, and does it transfer?
More capable models should also become better judges of their own limits.
Enlarge figure 
Enlarge chart 
