Prep 0.20.0 (#77)

* fix issue where recent save cover images would sometimes not load

* paraphrase prompt tweaks

* action_to_narration regenerate compatibility fixes

* sim suite add asnwer question instruction

* more sim suite tweaks

* refactor agent details display in agent bar

* visual agent progres (a1111 support)

* visual gen prompt tweaks

* openai compat client pass max_tokens

* world state sequential reinforcement max tokens tightened

* improve item names

* Improve item names

* attempt to remove "changed from.." notes when altering an existing character sheet

* prompt improvements for single character portraits

* visual agent progress

* fix issue where character.update wouldn't update long-term memory

* remove experimental flag for now

* add better instructions for updating existing character sheet

* background processing for agents, visual and tts

* fix selected voice not saving between restarts for elevenlabs

* lessen timeout

* clean up agent status logic

* conditional agent configs

* comfyui support

* visualization queue

* refactor visual styles, comfyui progress

* regen images
auto cover image assign
websocket handler plugin abstraction
agent websocket handler

* automatic1111 fixes
agent status and ready checks

* tweaks to character portrait prompt

* system prompt for visualize

* textgenwebui use temp smoothing on yi models

* comment out api key for now

* fixes issues with openai compat client for retaining api key and auto fixing urls

* update_reinforcment tweaks

* agent status emit from one place

* emit agent status as asyncio task

* remove debug output

* tts add openai support

* openai img gen support

* fix issue with confyui checkbox list not loading

* tts model selection for openai

* narrate_query include character sheet if character is referenced in query
improve visual character portrit generation prompt

* client implementation extra field support and runpod vllm client example

* relock

* fix issue where changing context length would cause next generation to error

* visual agent tweaks and auto gen character cover image in sim suite

* fix issue with readyness lock when there werent any clients defined

* load scene readiness fixes

* linting

* docs

* notes for the runpod vllm example

scenes/simulation-suite/templates/computer.jinja2

@@ -18,6 +18,7 @@ You must at least call one of the following functions:
 - set_player_persona
 - set_player_name
 - end_simulation
+- answer_question
 
 Set the player persona at the beginning of a new simulation or if the player requests a change.
 
@@ -52,7 +53,7 @@ Request: Computer, I want to experience a rollercoaster ride with a friend
 change_environment("theme park, riding a rollercoaster")
 set_player_persona("young female experiencing rollercoaster ride")
 set_player_name("Susanne")
-add_ai_character("a female friend of player")
+add_ai_character("a female friend of player named Sarah")
 ```
 
 Request: Computer, I want to experience the international space station
@@ -60,7 +61,7 @@ Request: Computer, I want to experience the international space station
 change_environment("international space station")
 set_player_persona("astronaut experiencing first trip to ISS")
 set_player_name("George")
-add_ai_character("astronaut")
+add_ai_character("astronaut named Henry")
 ```
 
 Request: Computer, remove the goblin and add an elven woman instead
@@ -77,19 +78,19 @@ change_ai_character("make skiing instructor older")
 Request: Computer, change my grandma to my grandpa
 ```simulation-stack
 remove_ai_character("grandma")
-add_ai_character("grandpa")
+add_ai_character("grandpa named Steven")
 ```
 
 Request: Computer, remove the skiing instructor and add my friend instead.
 ```simulation-stack
 remove_ai_character("skiing instructor")
-add_ai_character("player's friend")
+add_ai_character("player's friend named Tara")
 ```
 
 Request: Computer, replace the skiing instructor with my friend.
 ```simulation-stack
 remove_ai_character("skiing instructor")
-add_ai_character("player's friend")
+add_ai_character("player's friend named Lisa")
 ```
 
 Request: Computer, I want to end the simulation
@@ -102,6 +103,11 @@ Request: Computer, shut down the simulation
 end_simulation("simulation ended")
 ```
 
+Request: Computer, what do you know about the game of thrones?
+```simulation-stack
+answer_question("what do you know about the game of thrones?")
+```
+
 <|CLOSE_SECTION|>
 <|SECTION:TASK|>
 Respond with the simulation stack for the following request:

scenes/simulation-suite/templates/instructions.jinja2

@@ -26,6 +26,12 @@
                 {# change environment #}
                 {% set _ = processed.append(call) %}
             
+            {% elif call.strip().startswith("answer_question") %}
+                {# answert a query #}
+
+                {% set _ = agent_action("narrator", "action_to_narration", action_name="progress_story", narrative_direction="The computer calls the following function:\n"+call+"\nand answers the player's question.", emit_message=True) %}
+
+
             {% elif call.strip().startswith("set_player_persona") %}
                 {# treansform player #}
                 {% set _ = emit_status("busy", "Simulation suite altering user persona.", as_scene_message=True) %}
@@ -60,9 +66,10 @@
                 {% set _ = emit_status("busy", "Simulation suite adding character: "+character_name, as_scene_message=True) %}
                 {% set _ = debug("HOLODECK add npc", name=character_name)%}
                 {% set npc = agent_action("director", "persist_character", name=character_name, content=player_message.raw )%}
-                {% set _ = agent_action("world_state", "manager", action_name="add_detail_reinforcement", character_name=npc.name, question="Goal", instructions="Generate a goal for the character, based on the user's chosen simulation", interval=25, run_immediately=True) %}
+                {% set _ = agent_action("world_state", "manager", action_name="add_detail_reinforcement", character_name=npc.name, question="Goal", instructions="Generate a goal for "+npc.name+", based on the user's chosen simulation", interval=25, run_immediately=True) %}
                 {% set _ = debug("HOLODECK added npc", npc=npc) %}
                 {% set _ = processed.append(call) %}
+                {% set _ = agent_action("visual", "generate_character_portrait", character_name=npc.name) %}
             {% elif call.strip().startswith("remove_ai_character") %}
                 {# remove npc #}
 
@@ -80,12 +87,14 @@
                 {# change existing npc #}
 
                 {% set _ = emit_status("busy", "Simulation suite altering character.", as_scene_message=True) %}
-                {% set character_name = agent_action("creator", "determine_character_name", character_name=inject+" - what is the name of the character receiving the changes?", allowed_names=scene.npc_character_names) %}
+                {% set character_name = agent_action("creator", "determine_character_name", character_name=inject+" - what is the name of the character receiving the changes (before the change)?", allowed_names=scene.npc_character_names) %}
+
+                {% set character_name_after = agent_action("creator", "determine_character_name", character_name=inject+" - what is the name of the character receiving the changes (after the changes)?") %}
 
                 {% set npc = scene.get_character(character_name) %}
 
                 {% if npc %}
-                    {% set _ = emit_status("busy", "Changing "+character_name, as_scene_message=True) %}
+                    {% set _ = emit_status("busy", "Changing "+character_name+" -> "+character_name_after, as_scene_message=True) %}
                     {% set _ = debug("HOLODECK transform npc", npc=npc) %}
                     {% set character_attributes = agent_action("world_state", "extract_character_sheet", name=npc.name, alteration_instructions=player_message.raw)%}
                     {% set _ = npc.update(base_attributes=character_attributes) %}
@@ -93,15 +102,18 @@
                     {% set _ = npc.update(description=character_description) %}
                     {% set _ = debug("HOLODECK transform npc", attributes=character_attributes, description=character_description) %}
                     {% set _ = processed.append(call) %}
+                    {% if character_name_after != character_name %}
+                        {% set _ = npc.rename(character_name_after) %}
+                    {% endif %}
                 {% endif %}
             {% elif call.strip().startswith("end_simulation") %}
                 {# end simulation #}
                 {% set explicit_command = query_text_eval("has the player explicitly asked to end the simulation?", player_message.raw) %}
                 {% if explicit_command %}
                     {% set _ = emit_status("busy", "Simulation suite ending current simulation.", as_scene_message=True) %}
-                    {% set _ = agent_action("narrator", "action_to_narration", action_name="progress_story", narrative_direction="The computer ends the simulation, disolving the environment and all artifical characters, erasing all memory of it and finally returning the player to the inactive simulation suite.", emit_message=True) %}
+                    {% set _ = agent_action("narrator", "action_to_narration", action_name="progress_story", narrative_direction="The computer ends the simulation, disolving the environment and all artifical characters, erasing all memory of it and finally returning the player to the inactive simulation suite.List of artificial characters: "+(",".join(scene.npc_character_names))+". The player is also transformed back to their normal persona.", emit_message=True) %}
                     {% set _ = scene.sync_restore() %}
-                    {% set update_world_state = True %}
+                    {% set _ = agent_action("world_state", "update_world_state", force=True) %}
                     {% set simulation_reset = True %}
                 {% endif %}
             {% elif "(" in call.strip() %}
@@ -122,7 +134,7 @@
         {% set _ = emit_status("busy", "Simulation suite powering up.", as_scene_message=True) %}
         {% set _ = game_state.set_var("instr.simulation_started", "yes", commit=False) %}
         {% set _ = agent_action("narrator", "action_to_narration", action_name="progress_story", narrative_direction="Narrate the computer asking the user to state the nature of their desired simulation.", emit_message=False) %}
-        {% set _ = agent_action("narrator", "action_to_narration", action_name="paraphrase", narration="Please state your commands by addressing the computer by stating \"Computer,\" followed by an instruction.") %}
+        {% set _ = agent_action("narrator", "action_to_narration", action_name="passthrough", narration="Please state your commands by addressing the computer by stating \"Computer,\" followed by an instruction.") %}
 
         {# pin to make sure characters don't try to interact with the simulation #}
         {% set _ = agent_action("world_state", "manager", action_name="save_world_entry", entry_id="sim.quarantined", text="Characters in the simulation ARE NOT AWARE OF THE COMPUTER.", meta=make_dict(), pin=True) %}