fix: strip smart quotes from character example dialogue (#173) (#180)

* fix: strip smart quotes from character example dialogue (#173)

* fix: complete smart-quote pairs and normalize persist_character examples (#173 review)

* docs: correct stale verbatim wording and comment/log accuracy (#173 review)

CHANGELOG.yaml

@@ -11,7 +11,7 @@
   - "Visual Prompt Finalization: The Visualizer agent gained a Prompt Finalization settings tab defining post-processing actions (exact, fuzzy or regex match and replace, or an AI instruction) that rewrite image prompts right before they are sent to the image generation backend. Actions can target positive and/or negative prompts, be restricted to specific visual types, and be overridden per scene. Characters can define their own actions under World Editor → Characters → Visuals → Prompt Finalization, which run after the agent's. Reusable action sets are managed as a new 'Visual prompt finalizer' template type, including a shipped Ideogram JSON preset that converts the positive prompt into an Ideogram 4.0 structured JSON prompt. Prompt-only generation output is finalized as well, and a new FinalizePrompt node exposes the step to custom node graphs."
   - "Client Auto Retry: Clients can now automatically retry on response issues before you are notified — separate 0–5 retry sliders for empty responses, API rate limiting (HTTP 429, with progressively longer waits between attempts), and reasoning models that skip their reasoning tokens. A notification shows the retry progress with an abort option, and when retries run out (or a slider is at 0, the default) the usual generation error dialog appears."
   - "New Node Graph Events: Node graphs can now hook into more of this release's features. `agent.visual.prompt_finalize.before` / `.after` fire around visual prompt finalization (mutable prompts, and the `.before` finalizer list can be modified — they fire even with the agent setting disabled, so a graph can act as its own finalizer). `agent.creator.dialogue_examples.before` / `.after` fire around character example dialogue generation (inject instructions, or rewrite the generated examples - not emitted when a Fast-mode consolidated one-shot produces the examples itself). `agent.help.chat.before` / `.after` fire around help chat responses. Scene asset mutations gained `asset_deleted`, `scene.backdrop_changed`, `scene.cover_image_changed` and `character.cover_image_changed` alongside the existing `asset_saved`. All are documented in the node editor's Events reference."
-  - "Simplified Character Card Import: The character card import dialog gained an AI Generation section with individual toggles for each generation step — content context, description rewrite, attribute extraction, dialogue instructions, example dialogue, and story intent — plus Full/Minimal preset buttons to flip them all at once. Disabled steps fall back to the card's original data (the description and example dialogue import verbatim), so a minimal import needs no text generation at all and completes in a fraction of the time."
+  - "Simplified Character Card Import: The character card import dialog gained an AI Generation section with individual toggles for each generation step — content context, description rewrite, attribute extraction, dialogue instructions, example dialogue, and story intent — plus Full/Minimal preset buttons to flip them all at once. Disabled steps fall back to the card's original data (the description imports verbatim, example dialogue as-is with typographic quotes converted), so a minimal import needs no text generation at all and completes in a fraction of the time."
   - "Scene Browser Landing Page: The home screen was rebuilt into a full-page landing view, replacing the sidebar load panel. A file-tree Scene Library lists scene projects with cover thumbnails, save counts and per-save metadata, plus an asset and node-module summary row per expanded project; long save lists collapse behind a 'Show all' row and the list is filterable with click-to-load. Individual scene files and character cards can be deleted from the tree, and an entire scene project (saves, assets, nodes, history) can be deleted behind a type-the-project-name confirmation. Character cards display their card image and only actual card files are listed. Scene and character card import moved to a prominent drag-and-drop dropzone next to a Create-new-scene card, with the quick-load recent scene cards kept on top."
   - "Fast Character Creation: The Creator agent's settings gained a Character Creation section with a Fast mode that consolidates AI character generation into a single prompt instead of one prompt per aspect, covering the selected aspects in a single request. A Consolidate multi-select picks which aspects (name, description, attributes, dialogue instructions, example dialogue) the one-shot covers - unselected aspects keep their individual prompts - an 'Attribute templates' entry in the same list folds selected attribute world-state templates into the one-shot as per-attribute instructions instead of one prompt per template, a One-shot token budget slider (1024-8192, default 4096) controls the shared response budget, and a Fill in misses toggle re-runs an aspect's individual request when the consolidated response missed it, while a completely unparseable response is always a hard error. The consolidated prompt also applies the scene's writing style template when one is configured. All AI-assisted character creation paths (world editor, scene tools character introduction, character card import and node graphs) route through this single flow, with the split per-aspect flow remaining the default for older or smaller models. Off by default."
   - "Generate Character Node: Node graphs gained a Generate Character node (`agents/creator/GenerateCharacter`) that generates character data (name, description, attributes, dialogue instructions and/or example dialogue) following the creator agent's Character Creation settings. Which aspects to generate can be wired in, alongside the character name, guiding instructions, an existing description and generation options. The node only generates data - wire its outputs into the director Persist Character node, which now accepts pre-generated dialogue instructions and example dialogue, to add the character to the scene without generating twice. The `Create Character` module, which now builds its character through this node, gained matching `generate_dialogue_instructions` (default on - existing graphs calling the module now generate dialogue instructions, an extra prompt when Fast mode is off) and `generate_example_dialogue` (default off) inputs; the simulation suite enables both, so its add-character action now also generates example dialogue."
@@ -62,6 +62,7 @@
   - "Character Card Import: Attribute extraction no longer runs blind in the default (non-Fast) import mode — the character being profiled was missing from its own extraction prompt, so attributes were generated from the greeting and character book alone, without the character's description. Imported attributes now reflect the description."
   - "Character Creation: The director's 'Limit character attributes' setting now delivers the number of attributes it promises. The character's own name is written into the generated character sheet as a `Name` line, and it was counted against the limit — so every value delivered one attribute fewer than configured, and a limit of 1 produced no attributes at all. The name no longer costs a slot, in generated sheets as well as in sheets supplied to the Persist Character node."
   - "Fast Character Creation: The one-shot generation prompt no longer repeats the character's name as an attribute — the name is already generated as its own aspect, so a `Name` entry in the character sheet was duplication, and under a configured attribute limit it consumed one of the allowed lines. This applies at every value of the director's 'Limit character attributes' setting, including the default of 0: Fast mode no longer asks for a `Name` attribute, and one written anyway does not cost a slot."
+  - "Example Dialogue: Character example dialogue no longer keeps typographic quotes (“ ” „ ‘ ’ ‚). Talemate delimits spoken words with the straight quote, so an example line carrying fancy quotes rendered as narration instead of dialogue and was mis-chunked by dialogue parsing and text-to-speech. Typographic quotes are now replaced with their plain equivalents wherever example dialogue is written outside a generation — character card import, manual entry and edits in the world editor, examples supplied directly to the Persist Character node, and the creator's example dialogue generation. Generated text was already normalized at the client level, and now also covers the German-style low-9 opening quotes, so a `„…“` pair no longer collapses into a single unbalanced straight quote. Characters already saved with fancy quotes keep them until the affected line is saved again."
   - "Help Agent: Documentation lookups now find the page that actually answers the question. Every word counted the same, so asking about 'koboldcpp settings' returned five different agent settings pages and no KoboldCpp page at all — the common word decided the match and the rest were ties broken alphabetically. Distinctive words now count for far more than ubiquitous ones. Page length also buys far less rank: a longer description of a page used to be a strictly better one, every extra word another free chance to match, so describing a page more thoroughly made it surface for topics it only mentions in passing. Those matches are now diluted by how much the description covers, so a page that mentions a topic in passing no longer outranks the page about it."
 
 0.38.0:

docs/user-guide/character-card-import.md

@@ -117,7 +117,7 @@ By default, importing a character card runs several AI generation steps that enr
 
 - **Generate Dialogue Instructions**: AI generates acting instructions that guide how the character speaks and behaves.
 
-- **Generate Example Dialogue**: AI generates example dialogue for the character. When disabled, the card's original example dialogue is imported verbatim.
+- **Generate Example Dialogue**: AI generates example dialogue for the character. When disabled, the card's original example dialogue is imported as-is, with typographic quotes converted to straight quotes.
 
 - **Generate Story Intent**: AI generates the overall story intent for the scene. When the Director's auto-direct is enabled, this also covers scene type generation and scene intent setup.
 
@@ -180,7 +180,7 @@ Import an existing player character from another Talemate scene.
 
 - Character attributes (determined by AI analysis, if enabled)
 
-- Dialogue examples (generated from character card data, or imported verbatim if generation is disabled)
+- Dialogue examples (generated from character card data, or imported as-is if generation is disabled, with typographic quotes converted to straight quotes)
 
 - Character color (automatically assigned unique colors)
 

src/talemate/agents/creator/character.py

@@ -24,6 +24,7 @@ from talemate.game import focal
 from talemate.prompts import Prompt
 from talemate.prompts.response import ResponseSpec, StrictAnchorExtractor
 from talemate.util.data import parse_attribute_lines
+from talemate.util.strings import replace_smart_quotes
 from talemate.ux.schema import Condition
 from talemate.world_state.templates.content import GenerationOptions
 
@@ -263,13 +264,14 @@ def _normalize_example_dialogue_lines(
     - lines with an unrecognized name-shaped speaker prefix (another
       character's line) are dropped
     - anything else is prefixed with the final name
+    - typographic quotes are replaced with their ascii equivalents
     """
     known_prefixes = {
         p.lower() for p in [example_name, *(other_known_names or [])] if p and p.strip()
     }
     normalized = []
     for line in lines:
-        line = line.strip()
+        line = replace_smart_quotes(line).strip()
         if not line:
             continue
         speaker, colon, rest = line.partition(":")

src/talemate/agents/director/character_management.py

@@ -5,7 +5,12 @@ import structlog
 import talemate.instance as instance
 import talemate.agents.tts.voice_library as voice_library
 from talemate.agents.tts.schema import Voice
-from talemate.util import random_color, chunk_items_by_tokens, remove_substring_names
+from talemate.util import (
+    random_color,
+    chunk_items_by_tokens,
+    remove_substring_names,
+    replace_smart_quotes,
+)
 from talemate.util.data import trim_attributes
 from talemate.character import Character, set_voice, activate_character
 from talemate.status import LoadingStatus
@@ -638,6 +643,12 @@ class CharacterManagementMixin:
                 )
 
             if example_dialogue:
+                # generated examples arrive already normalized from the
+                # creator; pre-supplied ones (node graph literals, ws payload)
+                # skip generation, so this is where they get cleaned
+                example_dialogue = [
+                    replace_smart_quotes(example) for example in example_dialogue
+                ]
                 character.example_dialogue = example_dialogue
                 log.debug("persist_character", example_dialogue=example_dialogue)
 

src/talemate/character.py

@@ -352,7 +352,7 @@ class Character(pydantic.BaseModel):
         """
         Append a new example dialogue line.
         """
-        text = (example or "").strip()
+        text = util.replace_smart_quotes(example or "").strip()
         if not text:
             return
         self.example_dialogue.append(text)
@@ -363,7 +363,7 @@ class Character(pydantic.BaseModel):
         """
         if index < 0 or index >= len(self.example_dialogue):
             return
-        value = (text or "").strip()
+        value = util.replace_smart_quotes(text or "").strip()
         if not value:
             # empty string behaves like delete
             await self.remove_example_dialogue(index)

src/talemate/client/base.py

@@ -1984,12 +1984,7 @@ class ClientBase:
             self.end_request()
 
             if REPLACE_SMART_QUOTES:
-                response = (
-                    response.replace("“", '"')
-                    .replace("”", '"')
-                    .replace("‘", "'")
-                    .replace("’", "'")
-                )
+                response = util.replace_smart_quotes(response)
 
             time_end = time.time()
 

src/talemate/load/character_card.py

@@ -17,7 +17,12 @@ from talemate.exceptions import GenerationCancelled, LLMAccuracyError, UnknownDa
 from talemate.files import identify_character_card_spec
 from talemate.status import LoadingStatus
 from talemate.config import get_config
-from talemate.util import extract_metadata, select_best_texts_by_keyword, count_tokens
+from talemate.util import (
+    extract_metadata,
+    select_best_texts_by_keyword,
+    count_tokens,
+    replace_smart_quotes,
+)
 from talemate.util.colors import unique_random_colors
 from talemate.agents.base import DynamicInstruction
 from talemate.game.engine.nodes.registry import import_scene_node_definitions
@@ -1790,8 +1795,10 @@ def character_from_chara_data(data: dict) -> Character:
     if "color" in data:
         character.color = data["color"]
     if "mes_example" in data:
-        new_line_match = "\r\n" if "\r\n" in data["mes_example"] else "\n"
-        for message in data["mes_example"].split("<START>"):
+        # cards are hand-written; typographic quotes are common in them
+        mes_example = replace_smart_quotes(data["mes_example"])
+        new_line_match = "\r\n" if "\r\n" in mes_example else "\n"
+        for message in mes_example.split("<START>"):
             if message.strip(new_line_match):
                 character.example_dialogue.extend(
                     [m for m in message.split(new_line_match) if m]

src/talemate/util/strings.py

@@ -2,8 +2,30 @@
 
 __all__ = [
     "normalize_name",
+    "replace_smart_quotes",
 ]
 
+# Typographic quotes mapped to their ascii equivalents. Talemate uses the
+# straight `"` as the dialogue delimiter throughout (scene message rendering,
+# dialogue/exposition separation, tts markup), so smart quotes have to be
+# normalized away before text is stored or displayed.
+#
+# Both marks of a pair have to be mapped together: mapping only one of them
+# leaves a single unbalanced straight quote behind, which the dialogue parsers
+# then pair with whatever follows - worse than leaving the pair alone. The
+# low-9 marks are the opening quotes of the German-style `„…“` / `‚…‘`
+# convention, so they belong with the marks that close them.
+SMART_QUOTE_TRANSLATION = str.maketrans(
+    {
+        "“": '"',  # left double quotation mark
+        "”": '"',  # right double quotation mark
+        "„": '"',  # double low-9 quotation mark
+        "‘": "'",  # left single quotation mark
+        "’": "'",  # right single quotation mark
+        "‚": "'",  # single low-9 quotation mark
+    }
+)
+
 
 def normalize_name(raw: str | None, max_length: int) -> str | None:
     """
@@ -25,3 +47,16 @@ def normalize_name(raw: str | None, max_length: int) -> str | None:
     if not trimmed:
         return None
     return trimmed[:max_length]
+
+
+def replace_smart_quotes(text: str) -> str:
+    """
+    Replace typographic quotes with their ascii equivalents.
+
+    Args:
+        text: The input string.
+
+    Returns:
+        The string with smart quotes replaced.
+    """
+    return text.translate(SMART_QUOTE_TRANSLATION)

src/talemate/world_state/manager.py

@@ -17,6 +17,7 @@ from talemate.character import (
 )
 from talemate.instance import get_agent
 from talemate.emit import emit
+from talemate.util import replace_smart_quotes
 from talemate.world_state import (
     ContextPin,
     ManualContext,
@@ -502,8 +503,10 @@ class WorldStateManager:
 
         if example_dialogue:
             for idx, example in enumerate(example_dialogue):
+                example = replace_smart_quotes(example)
                 if not example.startswith(f"{character_name}:"):
-                    example_dialogue[idx] = f"{character_name}: {example}"
+                    example = f"{character_name}: {example}"
+                example_dialogue[idx] = example
 
         character.example_dialogue = example_dialogue
 

tests/test_character.py

@@ -235,6 +235,20 @@ class TestDialogueExamples:
         await c.add_example_dialogue(None)  # type: ignore[arg-type]
         assert c.example_dialogue == []
 
+    @pytest.mark.asyncio
+    async def test_add_example_dialogue_replaces_smart_quotes(self):
+        c = Character(name="Bob")
+        await c.add_example_dialogue("Bob: “Hello!” He waves. ‘Really.’")
+        assert c.example_dialogue == ["Bob: \"Hello!\" He waves. 'Really.'"]
+
+    @pytest.mark.asyncio
+    async def test_set_example_dialogue_item_replaces_smart_quotes(
+        self, make_character
+    ):
+        c = make_character()
+        await c.set_example_dialogue_item(0, "Alice: “New line”")
+        assert c.example_dialogue[0] == 'Alice: "New line"'
+
     @pytest.mark.asyncio
     async def test_set_example_dialogue_item_replaces_at_index(self, make_character):
         c = make_character()

tests/test_character_card_parsing.py

@@ -676,6 +676,16 @@ class TestCharacterFromCharaData:
         c = character_from_chara_data(data)
         assert "Bob: Hi!" in c.example_dialogue
 
+    def test_mes_example_replaces_smart_quotes(self):
+        data = {
+            "name": "Alice",
+            "mes_example": "<START>\nAlice: “Hi there.” She waves. ‘Really.’\n",
+        }
+        c = character_from_chara_data(data)
+        assert c.example_dialogue == [
+            "Alice: \"Hi there.\" She waves. 'Really.'",
+        ]
+
     def test_gender_field_persists_to_base_attributes(self):
         # 'gender' on Character is a read-only @property backed by
         # base_attributes; the loader writes to base_attributes so the

tests/test_creator_character.py

@@ -557,6 +557,26 @@ async def test_generate_character_unified_example_drops_bare_name_lines(creator)
     ]
 
 
+@pytest.mark.asyncio
+async def test_generate_character_unified_example_replaces_smart_quotes(creator):
+    async with MockClientContext():
+        client_responses.get().append(
+            unified_response(
+                name="Elena",
+                example_dialogue="Elena: “Hello.” She smiles.",
+            )
+        )
+        result = await creator.generate_character_unified(
+            CharacterGenerationRequest(
+                aspects=["name", "example_dialogue"],
+                name="the tall woman",
+                content="A healer arrives.",
+            )
+        )
+
+    assert result.example_dialogue == ['Elena: "Hello." She smiles.']
+
+
 @pytest.mark.asyncio
 async def test_generate_character_unified_example_speaker_heuristics(creator):
     # multi-word name-shaped prefixes (another character's line) are dropped -

tests/test_director_character_management.py

@@ -502,6 +502,29 @@ class TestPersistCharacterExampleDialogue:
         assert prompt_kinds(scene.mock_client) == [KIND_DIALOGUE_INSTRUCTIONS]
         assert not character.example_dialogue
 
+    @pytest.mark.asyncio
+    async def test_supplied_examples_are_smart_quote_normalized(self, scene, director):
+        # pre-supplied examples (node graph literals, ws payload) skip the
+        # generation step entirely, so this is the only place they are cleaned
+        async with MockClientContext():
+            client_responses.get().append("speaks softly")
+            character = await director.persist_character(
+                PersistCharacterRequest(
+                    name="Isolde",
+                    content="A travelling scribe",
+                    determine_name=False,
+                    generate_attributes=False,
+                    description="Already described.",
+                    narrate_entry=False,
+                    example_dialogue=["Isolde: “Ink and patience.” She nods."],
+                )
+            )
+
+        assert character is not None
+        # no example dialogue prompt ran - the supplied list was used as-is
+        assert prompt_kinds(scene.mock_client) == [KIND_DIALOGUE_INSTRUCTIONS]
+        assert character.example_dialogue == ['Isolde: "Ink and patience." She nods.']
+
     def test_payload_fields_match_persist_character_signature(self):
         """Every field the websocket payload exposes must be a
         PersistCharacterRequest field — guards backend/frontend parity when

tests/test_strings.py

@@ -0,0 +1,49 @@
+"""Unit tests for talemate.util.strings."""
+
+import pytest
+
+from talemate.util import replace_smart_quotes
+from talemate.util.dialogue import separate_dialogue_from_exposition
+
+
+class TestReplaceSmartQuotes:
+    @pytest.mark.parametrize(
+        "text,expected",
+        [
+            ("“Hello”", '"Hello"'),
+            ("‘Hello’", "'Hello'"),
+            (
+                "Alice: “Hi there.” She waves. ‘Really.’",
+                "Alice: \"Hi there.\" She waves. 'Really.'",
+            ),
+            # mixed straight / typographic, as seen in issue #173
+            (
+                'Alice: "When the equations refuse to sing.”',
+                'Alice: "When the equations refuse to sing."',
+            ),
+            # German-style low-9 opening quotes pair with the marks above
+            ("Alice: „Hallo“", 'Alice: "Hallo"'),
+            ("Alice: ‚Hallo‘", "Alice: 'Hallo'"),
+        ],
+    )
+    def test_replaces(self, text, expected):
+        assert replace_smart_quotes(text) == expected
+
+    def test_leaves_straight_quotes_untouched(self):
+        text = 'Alice: "Hello!" She waves. It\'s fine.'
+        assert replace_smart_quotes(text) == text
+
+    def test_handles_empty_input(self):
+        assert replace_smart_quotes("") == ""
+
+    def test_quote_pairs_stay_balanced_for_the_dialogue_parser(self):
+        # mapping only one mark of a pair would leave a single unbalanced
+        # straight quote, and the parser would then read the trailing
+        # narration as dialogue
+        line = "Alice: „Hallo“ she said, then waved."
+        chunks = separate_dialogue_from_exposition(replace_smart_quotes(line))
+        assert [(c.type, c.text) for c in chunks] == [
+            ("exposition", "Alice: "),
+            ("dialogue", '"Hallo"'),
+            ("exposition", " she said, then waved."),
+        ]

tests/test_world_state_manager.py

@@ -287,6 +287,18 @@ class TestUpdateCharacterScalars:
             "Alice: goodbye",
         ]
 
+    @pytest.mark.asyncio
+    async def test_update_character_actor_replaces_smart_quotes(self, scene, manager):
+        ch = make_actor(scene, "Alice")
+        await manager.update_character_actor(
+            "Alice",
+            example_dialogue=["Alice: “hi”", "“hello!” ‘really’"],
+        )
+        assert ch.example_dialogue == [
+            'Alice: "hi"',
+            "Alice: \"hello!\" 'really'",
+        ]
+
 
 # ---------------------------------------------------------------------------
 # rename_character_folder