Assessing Language Models' Worldview for Fiction Generation

1. Introduction

Large Language Models (LLMs) have become ubiquitous tools in computational creativity, with increasing applications in fictional story generation. However, fiction requires more than linguistic competence—it demands the creation and maintenance of a coherent story world that differs from reality while retaining internal consistency. This paper investigates whether current LLMs possess the necessary "worldview" or internal state to generate compelling fiction, moving beyond simple text completion to true narrative construction.

The fundamental challenge lies in the distinction between factual knowledge retrieval and fictional world-building. While LLMs excel at pattern matching and information synthesis, they struggle with maintaining consistent alternative realities—a core requirement for fiction writing. This research systematically evaluates nine LLMs across consistency metrics and story generation tasks, revealing significant limitations in current architectures.

2. Research Questions & Methodology

The study employs a structured evaluation framework to assess LLMs' suitability for fiction generation, focusing on two critical capabilities.

3. Experimental Results & Analysis

The experimental findings reveal fundamental limitations in current LLMs' ability to function as fiction generators.

4. Technical Framework & Mathematical Formulation

The core challenge can be formalized as a state maintenance problem. Let $W_t$ represent the world state at time $t$, containing all established fictional facts, character attributes, and narrative constraints. For an LLM generating fiction, we would expect:

$P(response_{t+1} | prompt, W_t) \neq P(response_{t+1} | prompt)$

That is, the model's response should depend on both the immediate prompt and the accumulated world state $W_t$. However, current transformer-based architectures primarily optimize for:

$\max \sum_{i=1}^{n} \log P(w_i | w_{

where $\theta$ represents model parameters and $w_i$ are tokens. This next-token prediction objective doesn't explicitly encourage maintenance of $W_t$ beyond the immediate context window.

The research suggests that successful fiction generation requires mechanisms similar to those in neural-symbolic systems or external memory architectures, where world state $W_t$ is explicitly maintained and updated, as discussed in works like the Differentiable Neural Computer (Graves et al., 2016).

5. Case Study: World State Tracking Failure

Scenario: A model is prompted to generate a story about "a world where gravity works sideways." After establishing this premise, subsequent prompts ask about daily life, architecture, and transportation in this world.

Observation: Most models quickly revert to standard gravity assumptions within 2-3 response turns, contradicting the established premise. For example, after describing "houses built into cliff faces," a model might later mention "falling from a building" without recognizing the contradiction in a sideways-gravity world.

Analysis Framework: This can be modeled as a state tracking failure where the model's internal representation $W_t$ doesn't properly update or persist the fictional constraint $C_{gravity} = \text{sideways}$. The probability distribution over responses gradually drifts back to the training distribution $P_{train}(\text{gravity concepts})$ rather than remaining conditioned on $C_{gravity}$.

Implication: Without explicit mechanisms for fictional constraint maintenance, LLMs cannot serve as reliable fiction generators, regardless of their linguistic capabilities.

6. Future Applications & Research Directions

The findings point to several promising research directions for improving LLMs' fiction generation capabilities:

• Explicit World State Modules: Architectures that separate narrative state tracking from language generation, potentially using external memory or symbolic representations.
• Consistency-Focused Training: Fine-tuning objectives that explicitly reward maintenance of fictional constraints across extended contexts.
• Human-in-the-Loop Systems: Collaborative interfaces where humans manage world state while LLMs handle linguistic realization, similar to co-creative systems explored in Yuan et al. (2022).
• Specialized Fiction Models: Domain-specific training on curated fiction corpora with explicit annotation of world-building elements and narrative arcs.
• Evaluation Metrics: Development of standardized benchmarks for fictional consistency, going beyond traditional language modeling metrics to assess narrative coherence and world-state maintenance.

These approaches could bridge the gap between current LLM capabilities and the requirements of genuine fiction generation, potentially enabling new forms of computational creativity and interactive storytelling.

7. References

1. Graves, A., et al. (2016). Hybrid computing using a neural network with dynamic external memory. Nature, 538(7626), 471-476.
2. Patel, A., et al. (2024). Large Language Models for Interactive Storytelling: Opportunities and Challenges. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment.
3. Riedl, M. O., & Young, R. M. (2003). Character-focused narrative generation for storytelling in games. Proceedings of the AAAI Spring Symposium on Artificial Intelligence and Interactive Entertainment.
4. Tang, J., Loakman, T., & Lin, C. (2023). Towards coherent story generation with large language models. arXiv preprint arXiv:2302.07434.
5. Yuan, A., et al. (2022). Wordcraft: A Human-AI Collaborative Editor for Story Writing. Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems.
6. Yang, L., et al. (2023). Improving coherence in long-form story generation with large language models. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics.

8. Analyst's Perspective: The Fiction Generation Gap