Popper Framework
An open-source framework for computational skepticism and AI validation. Named after philosopher Karl Popper, this framework embraces the scientific method by rigorously examining evidence both for and against AI systems.
"We can never be completely certain, but we can be confident in what we've tried to falsify."
About the Project
The Popper framework is an open-source platform for systematic AI validation and computational skepticism. Rather than claiming models are "correct," Popper establishes a systematic approach to identifying strengths, weaknesses, biases, and inconsistencies in AI systems through an ecosystem of specialized validation agents orchestrated by a central verification layer.
Led by Professor Nik Bear Brown, PhD, MBA, this experimental project emphasizes learning by building, inviting contributors to discover what approaches actually work in practice rather than claiming to have definitive solutions.
Philosophical Foundations
The Popper framework is built on Karl Popper's revolutionary principle that scientific theories can never be proven "true" – they can only be corroborated through rigorous testing that fails to falsify them. In Popper's words, "Good tests kill flawed theories; we remain alive to guess again."
We apply this philosophy to AI validation through:
Balanced Evidence Assessment
Methodically gather and evaluate evidence both supporting and challenging AI systems.
Conjecture and Refutation
Propose potential strengths and weaknesses, then test them systematically.
Critical Rationalism
Subject all claims to rigorous scrutiny, regardless of source or confidence.
Scientific Skepticism
Embrace doubt as the path to reliable knowledge while recognizing corroborating evidence.
The Agent Ecosystem
The Popper framework organizes specialized agents into classes, each focused on different aspects of AI validation:
Data Validation Agents
Examine whether datasets accurately represent reality.
Bias Detection Agents
Identify and mitigate various forms of bias in AI systems.
Explainability Agents
Make AI systems more transparent and interpretable.
Probabilistic Reasoning Agents
Evaluate uncertainty and probabilistic reasoning in AI.
Adversarial Agents
Test AI robustness through controlled attacks.
RL Validation Agents
Evaluate reinforcement learning systems for reliability and ethics.
Visualization Agents
Create interfaces between AI systems and human users.
Falsification Agents
Actively seek to disprove AI claims and identify limitations.
Graph-Based Reasoning Agents
Analyze relationships and dependencies in AI knowledge structures.
Causal Inference Agents
Examine and verify causal relationships in AI systems.
The Popper Orchestration Layer
At the heart of the framework is the Popper orchestration layer, which coordinates the activities of specialized agents to systematically evaluate AI systems:
Cross-Agent Validation
Testing approaches to identifying when different validation agents reach contradictory conclusions.
Dynamic Task Allocation
Exploring approaches to distributing validation resources based on changing priorities.
Pattern Recognition
Experiments with identifying connections across seemingly unrelated validation findings.
Decision Optimization
Exploring methodologies for translating validation insights into appropriate system improvements.
Continuous Learning
Testing how the entire validation framework might improve over time.
While AI technology evolves rapidly, the Popper framework is designed to adapt and learn. Our open-source approach emphasizes transparency, allowing contributors to understand the reasoning behind each component, challenge assumptions, and discover through experimentation which approaches yield the most valuable insights into AI validity.
The Popper project offers educational resources through videos, documentation, and collaborative development opportunities. We invite you to join us in this experimental journey of building and learning together.
Key Projects
Critical Evidence Framework
A balanced system for gathering, evaluating, and weighing evidence both supporting and challenging AI claims.
Causal Inference Pipeline
A comprehensive toolkit for rigorous causal analysis in AI systems, from DAG construction to sensitivity analysis.
Multi-Model Verification
A framework for cross-validating outputs across multiple AI models to identify consistencies and discrepancies.
Graph-Based Knowledge Validation
Tools for analyzing and validating the structure and relationships in AI knowledge representations.
Comprehensive Data Evaluation
A system for assessing the strengths and limitations of training and evaluation data.
Explanation Assessment
Tools for evaluating the quality, completeness, and faithfulness of AI-generated explanations.
Bias Detection Suite
A collection of techniques for identifying and measuring various forms of bias in AI systems.
Probabilistic Calibration
Tools for evaluating and improving the calibration of confidence estimates in AI predictions.
Contributing to Popper
We welcome contributions from the community! The Popper framework is an educational experiment designed to evolve through collaborative learning and development.
Develop New Agents
Create specialized agents for novel validation approaches.
Improve Existing Agents
Enhance the effectiveness of current validation techniques.
Benchmark Against Real Cases
Test Popper against known AI failures and successes.
Document Best Practices
Share what works and what doesn't in AI validation.
Integrate With AI Systems
Build connectors to popular AI frameworks and models.