Mythology

Mythology is an open research initiative dedicated to auditing outdated evaluation frameworks and developing state-of-the-art benchmarks built for the agentic era. Inspired by EleutherAI’s EvalEval initiative, Mythology began as an internal research initiative to fix how the industry measures machine intelligence.

The nature of AI has fundamentally shifted since the paradigms introduced by GPT-3. Modern models are complex, hierarchical cascades—networks of interconnected systems where an internal chain-of-thought and verification loops happen dynamically behind the scenes. Forcing these new-era models into legacy benchmarks is pointless. Traditional parameters like temperature have become increasingly obsolete as models natively negotiate their own reasoning paths, heavily defined during training runtime (Roo-Code #6965).

Core Methodology Focus

We are actively redesigning the evaluation ecosystem by addressing three systemic vulnerabilities:

• Unfit Evaluation Frameworks: Pre-2023 testing suites are built around rigid, static tasks. They are fundamentally unsuited for modern LLMs as well as advanced audio, video, and multi-modal models that display complex, hyper-pseudo-conscious agentic behaviors.
• Benchmark-Centric Optimization: Driven by a fixation on public leaderboards, modern model architectures—as demonstrated prominently by frontier systems like LLaMa-4—are frequently optimized explicitly to maximize scores and resist known evaluation attacks, masking real-world flaws.
• Dishonest Pre-training: Closed-source training pipelines obscure what truly happens inside pre-training loops. While explicit dataset contamination is rarely confessed openly, brilliant satirical commentary like "Rethinking Benchmark Contamination" cuts right to an underlying, honest truth: the industry operates on an open secret where models are frequently fed near-perfect instances of the test criteria to artificially inflate public performance.

Initiative Roadmap

Mythology is structured around a deliberate, high-impact release schedule designed to transition static AI evaluation into a living ecosystem:

1. Phase 1

   Initial Launch

   Establishing the initiative ecosystem and onboarding founding collaborators to baseline the core evaluation infrastructure.

2. Phase 2

   2-Part Paper Series

   Publishing targeted research analyzing legacy benchmark obsolescence, testing modern models and revelating vectors in existing benchmarks.

3. Phase 3

   Annual Evaluation Cycle

   Transitioning Mythology into a persistent, yearly benchmarking release framework that scales alongside emerging models.

Collaborations & Support

Mythology is driven by our founding collaborators, and we are actively accepting new individuals who meet our technical criteria.