From eef03e671ed7b25ffac1b420aeb9366ebe3f83ca Mon Sep 17 00:00:00 2001 From: Nikhil <118773738+pablohashescobar@users.noreply.github.com> Date: Fri, 8 Aug 2025 17:33:05 +0530 Subject: [PATCH] [WEB-4238] feat: event stream implementation (#3684) * Pgtrigger and Outbox prototype * feat: add event stream app with models and migrations - Introduced the event_stream app with initial models: Outbox and IssueProxy. - Implemented a PostgreSQL trigger for logging updates to issues. - Added necessary migrations and updated settings to include the new app. * refactor: migrate event publisher to event stream - Updated imports in test files to reflect the new event_stream app structure. - Removed the event_publisher app, including its models, migrations, and tests. - Introduced MemorySafeOutboxEventListener and related service for handling outbox events. - Added management command for starting the outbox event listener with memory safety. - Implemented PostgreSQL triggers for outbox notifications in the new event_stream app. * refactor: enhance issue serializer and event stream models - Updated IssueSerializer and IssueCreateSerializer to manage assignees and labels more efficiently by calculating additions and removals instead of deleting all and re-adding. - Introduced new proxy models for CycleIssue and ModuleIssue with corresponding PostgreSQL triggers for event handling. - Refactored Outbox model to include entity type and ID for better event tracking. - Removed outdated IssueProxy model and its associated triggers, streamlining the event stream architecture. * chore: update base requirements for OpenSearch and add pika for RabbitMQ - Retained django-opensearch-dsl version 0.7.0 in requirements. - Added pika version 1.3.2 for RabbitMQ integration. * refactor: update import paths for event stream listener - Changed import statements in test_competitive_listeners.py, service.py, and listen_outbox_events.py to reflect the new module structure by removing the 'apiserver' prefix. * feat: implement PostgreSQL LISTEN/NOTIFY listener for outbox events - Added DatabaseConnection class to manage psycopg3 connections for LISTEN/NOTIFY. - Introduced NotificationListener class to handle PostgreSQL NOTIFY messages and dispatch them to registered handlers. - Updated Command class to utilize NotificationListener for listening to outbox events, replacing the previous restartable listener implementation. - Improved error handling and logging throughout the listener process. * feat: enhance event processing with advisory locks - Implemented advisory lock mechanism to prevent concurrent processing of events. - Added methods to acquire and release advisory locks in the DatabaseConnection class. - Updated root_handler to fetch complete outbox data and mark events as processed only if all handlers succeed. - Improved error handling and logging for event processing. * fix: improve event logging and error handling in NotificationListener - Updated event logging to include event ID for better traceability. - Enhanced error handling in the command's handle method to log exceptions and maintain robustness during event listening. * feat: add memory monitoring and auto-restart functionality to event listener - Introduced MemoryMonitor class to track memory usage and event processing counts. - Implemented automatic restarts for the NotificationListener based on memory and event limits. - Enhanced command line options to configure memory limits, event limits, and memory check intervals. - Improved logging for memory stats and event processing to aid in monitoring and debugging. * feat: update local and production settings for event logging - Introduced logging configuration for 'plane.event_stream' in both local and production settings to enhance event logging capabilities. * chore: Implement async outbox polling mechanism with memory monitoring and event handling. Add management command for outbox poller and create necessary database migrations for outbox triggers and indexes. * docs: Add comprehensive README for Event Stream system detailing architecture, components, configuration, usage, and performance tuning. * feat: Enhance MongoDB integration with a singleton connection manager and implement outbox cleanup task for migrating records from PostgreSQL to MongoDB * feat: Add asyncio support for tests and new test files for outbox polling * Merge branch 'event-stream' of github.com:makeplane/plane-ee into event-stream * refactor: Update event types in outbox triggers for cycle, issue, and module models; add IssueComment, IssueLink, and IssueRelation proxies with corresponding triggers * refactor: Update import paths for JsonFormatter and consolidate OpenFeature imports; add unit tests for OutboxPoller and related classes * refactor: Replace all_objects with objects in Outbox queries and update test fixtures for outbox records * feat: Add daily outbox cleaner task to Celery and update type hint for MongoDB collection * refactor: Enhance memory monitoring and outbox polling logic with improved logging and delay handling * feat: Implement outbox triggers for issue, issue attachment, issue link, and issue relation models with dynamic event type determination * chore: Remove obsolete files related to multiple listeners and competitive processing tests * Delete service.py * feat: Add outbox poller script to manage database migrations and process events with configurable parameters * refactor: Optimize EpicCreateSerializer to handle initiative, assignee, and label updates with conflict management and improved logic for adding/removing related entities * feat: Update issue proxy triggers for outbox events - Removed the existing 'issue_outbox_update' trigger from the IssueProxy model. - Added a new 'issue_outbox_update' trigger with enhanced logic to handle conversion events when the issue type changes. - The new trigger captures updates and soft deletes, inserting appropriate events into the outbox. * feat: Refactor issue proxy triggers and add new outbox event handling - Removed and replaced existing triggers for IssueProxy, IssueAssigneeProxy, IssueLabelProxy, CycleIssueProxy, and ModuleIssueProxy. - Introduced new triggers to handle outbox events for various issue-related actions, including creation, updates, and deletions. - Enhanced logic for event type determination based on issue type, ensuring accurate event handling in the outbox. * refactor: Update serializers to use ID fields for assignees and labels - Changed the handling of assignees and labels in IssueCreateSerializer and EpicCreateSerializer to use ID fields instead of object references. - Improved logic for determining assignees and labels to add or remove by using lists of IDs. - Enhanced code readability with comments explaining the changes. * feat: Enhance issue event handling in outbox triggers - Updated triggers for IssueProxy, IssueAssigneeProxy, and IssueLabelProxy to determine event types based on issue type, including handling for epics. - Introduced filtered data for outbox events to exclude description fields, improving data integrity and reducing unnecessary payload size. - Enhanced logic for detecting changes during updates, ensuring only relevant changes are captured and processed. * feat: implement connection pooling for outbox poller - Introduced DatabaseConnectionPool class to manage async connection pooling using psycopg_pool. - Enhanced outbox polling logic to utilize connection pooling for improved performance and resource management. - Added health check and statistics retrieval methods for the connection pool. - Updated outbox model and migration to include claimed_at field for better event tracking. - Refactored tests to validate new connection pooling functionality. * refactor: move MongoConnection to a new settings module and update imports - Created a new mongo.py file to define the MongoConnection class for managing MongoDB connections. - Updated the import path in outbox_cleaner.py to reference the new MongoConnection location. * refactor: update import path for MongoConnection in outbox_cleaner.py * chore: remove scout_apm from production settings and requirements - Deleted scout_apm from the installed apps in production.py. - Removed scout-apm dependency from base.txt requirements. * feat: add claimed_at field to outbox events and update requirements - Added claimed_at field to the outbox event model for enhanced event tracking. - Updated outbox poller to handle claimed_at in event processing. - Included django-pgtrigger in base.txt requirements for database trigger management. * feat: enhance memory monitoring in outbox poller - Updated MemoryMonitor to signal for restarts instead of exiting on memory limit exceedance. - Added methods to check for restart requests and wait for restart signals. - Improved outbox poller logic to handle memory monitoring more gracefully during processing. - Ensured all claimed rows are processed before initiating a restart. * feat: implement graceful shutdown handling in outbox poller - Added GracefulShutdownHandler class to manage shutdown signals (SIGTERM, SIGINT, SIGQUIT). - Integrated shutdown handling into the outbox poller to allow for graceful exits during processing. - Updated polling logic to check for shutdown requests and clean up resources accordingly. - Enhanced command help text to reflect new signal handling capabilities. * Enhance outbox event handling by adding workspace and project IDs - Updated the outbox cleaner to include `workspace_id` and `project_id` in the deletion process. - Modified the outbox poller to handle new fields in event processing. - Adjusted models and migrations to support the new fields in the outbox table. - Updated tests to ensure proper handling of workspace and project IDs in outbox records. * Merge branch 'preview' of github.com:makeplane/plane-ee into event-stream * feat: add initiator_id field to Outbox model and update migration dependencies * feat: add initiator_id to Outbox insertions across event stream models - Updated Outbox insert statements in various models to include initiator_id. - Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure proper tracking of the user initiating changes. - Enhanced event handling for soft deletes and regular updates to maintain consistency in data tracking. * feat: include initiator_id in OutboxEvent model and database interactions - Added initiator_id field to the OutboxEvent model for enhanced tracking. - Updated database queries in outbox_poller.py to include initiator_id in insertions. - Ensured consistency in data handling across event stream components. * refactor: update handle_row function to use OutboxEvent model - Changed the parameter type of handle_row from Dict to OutboxEvent for better type safety. - Updated logging to utilize the to_dict method of OutboxEvent for consistent event data representation. * feat: enhance Outbox model with initiator type and update related logic - Added initiator_type field to the Outbox model to track the type of event initiator. - Introduced InitiatorTypes enum for better clarity and management of initiator types. - Updated OutboxEvent model and related methods to include initiator_type for consistent event data representation. - Adjusted database interactions across event stream models to accommodate the new initiator_type field. * feat: add initiator_type to Outbox insertions across event stream models - Updated Outbox insert statements in various models to include initiator_type for better tracking of event initiators. - Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure consistent handling of initiator_type during event processing. - Enhanced event handling for both soft deletes and regular updates to maintain data integrity and tracking. * feat: update event stream triggers to use After timing for improved consistency - Changed trigger timing from Before to After for various event stream models including CycleIssueProxy, IssueProxy, and IssueAssigneeProxy to ensure that outbox updates reflect the final state of the entities. - Enhanced logic in triggers to include previous attributes for better tracking of changes during updates. - Adjusted related logic in IssueLabelProxy and IssueCommentProxy to maintain consistency across event handling. * feat: enhance bulk update task with transaction management and initiator type - Wrapped bulk creation of issue relations and parent ID updates in a transaction to ensure atomicity. - Set the initiator type to 'SYSTEM.IMPORT' for both bulk creation and updates to improve tracking of operations. - Improved database interaction by using a connection cursor for executing SQL commands. --------- Co-authored-by: Dheeraj Kumar Ketireddy --- .../bin/docker-entrypoint-outbox-poller.sh | 15 + apps/api/plane/api/serializers/issue.py | 39 +- apps/api/plane/api/views/issue.py | 3 - apps/api/plane/app/serializers/issue.py | 42 +- .../0004_application_is_mentionable.py | 6 +- apps/api/plane/bgtasks/data_import_task.py | 8 +- apps/api/plane/celery.py | 5 + apps/api/plane/db/models/issue.py | 3 + apps/api/plane/db/mongodb.py | 25 - apps/api/plane/ee/serializers/app/epic.py | 143 +- .../ee/views/app/issue/bulk_operations.py | 15 +- apps/api/plane/event_stream/README.md | 695 ++++++++ apps/api/plane/event_stream/__init__.py | 1 + apps/api/plane/event_stream/apps.py | 5 + .../plane/event_stream/bgtasks/__init__.py | 0 .../event_stream/bgtasks/outbox_cleaner.py | 189 ++ .../plane/event_stream/management/__init__.py | 1 + .../management/commands/__init__.py | 1 + .../management/commands/outbox_poller.py | 789 +++++++++ .../event_stream/migrations/0001_initial.py | 217 +++ .../plane/event_stream/migrations/__init__.py | 0 .../api/plane/event_stream/models/__init__.py | 4 + apps/api/plane/event_stream/models/cycle.py | 102 ++ apps/api/plane/event_stream/models/issue.py | 1538 +++++++++++++++++ apps/api/plane/event_stream/models/module.py | 92 + apps/api/plane/event_stream/models/outbox.py | 267 +++ apps/api/plane/payment/flags/provider.py | 4 +- apps/api/plane/settings/common.py | 2 + apps/api/plane/settings/local.py | 7 +- apps/api/plane/settings/mongo.py | 118 ++ apps/api/plane/settings/production.py | 7 +- .../bulk_update_issue_relations_task.py | 19 +- apps/api/plane/tests/conftest.py | 11 +- .../contract/app/test_issue_duplicate.py | 1 - .../unit/bg_tasks/test_outbox_cleaner.py | 226 +++ .../api/plane/tests/unit/commands/__init__.py | 0 .../tests/unit/commands/test_outbox_poller.py | 536 ++++++ apps/api/plane/utils/url.py | 13 +- apps/api/pytest.ini | 7 +- apps/api/requirements/base.txt | 12 +- apps/api/requirements/test.txt | 3 +- 41 files changed, 5049 insertions(+), 122 deletions(-) create mode 100755 apps/api/bin/docker-entrypoint-outbox-poller.sh delete mode 100644 apps/api/plane/db/mongodb.py create mode 100644 apps/api/plane/event_stream/README.md create mode 100644 apps/api/plane/event_stream/__init__.py create mode 100644 apps/api/plane/event_stream/apps.py create mode 100644 apps/api/plane/event_stream/bgtasks/__init__.py create mode 100644 apps/api/plane/event_stream/bgtasks/outbox_cleaner.py create mode 100644 apps/api/plane/event_stream/management/__init__.py create mode 100644 apps/api/plane/event_stream/management/commands/__init__.py create mode 100644 apps/api/plane/event_stream/management/commands/outbox_poller.py create mode 100644 apps/api/plane/event_stream/migrations/0001_initial.py create mode 100644 apps/api/plane/event_stream/migrations/__init__.py create mode 100644 apps/api/plane/event_stream/models/__init__.py create mode 100644 apps/api/plane/event_stream/models/cycle.py create mode 100644 apps/api/plane/event_stream/models/issue.py create mode 100644 apps/api/plane/event_stream/models/module.py create mode 100644 apps/api/plane/event_stream/models/outbox.py create mode 100644 apps/api/plane/settings/mongo.py create mode 100644 apps/api/plane/tests/unit/bg_tasks/test_outbox_cleaner.py create mode 100644 apps/api/plane/tests/unit/commands/__init__.py create mode 100644 apps/api/plane/tests/unit/commands/test_outbox_poller.py diff --git a/apps/api/bin/docker-entrypoint-outbox-poller.sh b/apps/api/bin/docker-entrypoint-outbox-poller.sh new file mode 100755 index 0000000000..0bf7ba92fa --- /dev/null +++ b/apps/api/bin/docker-entrypoint-outbox-poller.sh @@ -0,0 +1,15 @@ +#!/bin/bash +set -e + +python manage.py wait_for_db +# Wait for migrations +python manage.py wait_for_migrations + + +# Run the processes +python manage.py outbox_poller \ + --memory-limit ${OUTBOX_POLLER_MEMORY_LIMIT_MB:-512} \ + --interval-min ${OUTBOX_POLLER_INTERVAL_MIN:-0.25} \ + --interval-max ${OUTBOX_POLLER_INTERVAL_MAX:-10} \ + --batch-size ${OUTBOX_POLLER_BATCH_SIZE:-250} \ + --memory-check-interval ${OUTBOX_POLLER_MEMORY_CHECK_INTERVAL:-30} \ No newline at end of file diff --git a/apps/api/plane/api/serializers/issue.py b/apps/api/plane/api/serializers/issue.py index 99bab7f008..ab5141f37f 100644 --- a/apps/api/plane/api/serializers/issue.py +++ b/apps/api/plane/api/serializers/issue.py @@ -255,7 +255,22 @@ class IssueSerializer(BaseSerializer): updated_by_id = instance.updated_by_id if assignees is not None: - IssueAssignee.objects.filter(issue=instance).delete() + # Get the current assignees + current_assignees = IssueAssignee.objects.filter( + issue=instance + ).values_list("assignee_id", flat=True) + + # Get the assignees to add + assignees_to_add = list(set(assignees) - set(current_assignees)) + + # Get the assignees to remove + assignees_to_remove = list(set(current_assignees) - set(assignees)) + + # Delete the assignees to remove + IssueAssignee.objects.filter( + issue=instance, assignee_id__in=assignees_to_remove + ).delete() + try: IssueAssignee.objects.bulk_create( [ @@ -267,7 +282,7 @@ class IssueSerializer(BaseSerializer): created_by_id=created_by_id, updated_by_id=updated_by_id, ) - for assignee_id in assignees + for assignee_id in assignees_to_add ], batch_size=10, ignore_conflicts=True, @@ -276,7 +291,23 @@ class IssueSerializer(BaseSerializer): pass if labels is not None: - IssueLabel.objects.filter(issue=instance).delete() + # Get the current labels + current_labels = IssueLabel.objects.filter(issue=instance).values_list( + "label_id", flat=True + ) + + # Get the labels to add + labels_to_add = list(set(labels) - set(current_labels)) + + # Get the labels to remove + labels_to_remove = list(set(current_labels) - set(labels)) + + # Delete the labels to remove + IssueLabel.objects.filter( + issue=instance, label_id__in=labels_to_remove + ).delete() + + # Create the labels to add try: IssueLabel.objects.bulk_create( [ @@ -288,7 +319,7 @@ class IssueSerializer(BaseSerializer): created_by_id=created_by_id, updated_by_id=updated_by_id, ) - for label_id in labels + for label_id in labels_to_add ], batch_size=10, ignore_conflicts=True, diff --git a/apps/api/plane/api/views/issue.py b/apps/api/plane/api/views/issue.py index c1cb69fea5..939503619f 100644 --- a/apps/api/plane/api/views/issue.py +++ b/apps/api/plane/api/views/issue.py @@ -2,7 +2,6 @@ import json import re import uuid -import re # Django imports from django.core.serializers.json import DjangoJSONEncoder @@ -31,7 +30,6 @@ from rest_framework.response import Response # drf-spectacular imports from drf_spectacular.utils import ( extend_schema, - OpenApiParameter, OpenApiResponse, OpenApiExample, OpenApiRequest, @@ -102,7 +100,6 @@ from plane.utils.openapi import ( EXTERNAL_ID_PARAMETER, EXTERNAL_SOURCE_PARAMETER, ORDER_BY_PARAMETER, - SEARCH_PARAMETER, SEARCH_PARAMETER_REQUIRED, LIMIT_PARAMETER, WORKSPACE_SEARCH_PARAMETER, diff --git a/apps/api/plane/app/serializers/issue.py b/apps/api/plane/app/serializers/issue.py index 12d758869c..33171623c4 100644 --- a/apps/api/plane/app/serializers/issue.py +++ b/apps/api/plane/app/serializers/issue.py @@ -366,7 +366,21 @@ class IssueCreateSerializer(BaseSerializer): updated_by_id = instance.updated_by_id if assignees is not None: - IssueAssignee.objects.filter(issue=instance).delete() + # Here because of validate the assignee_ids are list of uuids + current_assignees = IssueAssignee.objects.filter( + issue=instance + ).values_list("assignee_id", flat=True) + + # Get the assignees to add and assignees to remove from both the current and the new assignees + assignees_to_add = list(set(assignees) - set(current_assignees)) + assignees_to_remove = list(set(current_assignees) - set(assignees)) + + # Delete the assignees to remove + IssueAssignee.objects.filter( + issue=instance, assignee_id__in=assignees_to_remove + ).delete() + + # Create the assignees to add try: IssueAssignee.objects.bulk_create( [ @@ -378,7 +392,7 @@ class IssueCreateSerializer(BaseSerializer): created_by_id=created_by_id, updated_by_id=updated_by_id, ) - for assignee_id in assignees + for assignee_id in assignees_to_add ], batch_size=10, ignore_conflicts=True, @@ -387,19 +401,37 @@ class IssueCreateSerializer(BaseSerializer): pass if labels is not None: - IssueLabel.objects.filter(issue=instance).delete() + # Here we get label instances as a list + current_label_ids = list( + IssueLabel.objects.filter(issue=instance).values_list( + "label_id", flat=True + ) + ) + + requested_label_ids = labels + + # Get the labels to add and labels to remove from both the current and the new labels + labels_to_add = list(set(requested_label_ids) - set(current_label_ids)) + labels_to_remove = list(set(current_label_ids) - set(requested_label_ids)) + + # Delete the labels to remove + IssueLabel.objects.filter( + issue=instance, label_id__in=labels_to_remove + ).delete() + + # Create the labels to add try: IssueLabel.objects.bulk_create( [ IssueLabel( - label_id=label_id, + label_id=label, issue=instance, project_id=project_id, workspace_id=workspace_id, created_by_id=created_by_id, updated_by_id=updated_by_id, ) - for label_id in labels + for label in labels_to_add ], batch_size=10, ignore_conflicts=True, diff --git a/apps/api/plane/authentication/migrations/0004_application_is_mentionable.py b/apps/api/plane/authentication/migrations/0004_application_is_mentionable.py index 475771a9df..9eb3211297 100644 --- a/apps/api/plane/authentication/migrations/0004_application_is_mentionable.py +++ b/apps/api/plane/authentication/migrations/0004_application_is_mentionable.py @@ -6,13 +6,13 @@ from django.db import migrations, models class Migration(migrations.Migration): dependencies = [ - ('authentication', '0003_application_website_applicationcategory_is_active'), + ("authentication", "0003_application_website_applicationcategory_is_active"), ] operations = [ migrations.AddField( - model_name='application', - name='is_mentionable', + model_name="application", + name="is_mentionable", field=models.BooleanField(default=False), ), ] diff --git a/apps/api/plane/bgtasks/data_import_task.py b/apps/api/plane/bgtasks/data_import_task.py index c3eac197d0..254fb524ff 100644 --- a/apps/api/plane/bgtasks/data_import_task.py +++ b/apps/api/plane/bgtasks/data_import_task.py @@ -1,10 +1,12 @@ +# Python imports +import logging + # Third party imports from celery import shared_task import requests -from django.db import transaction +from django.db import transaction, connection from django.conf import settings -import logging from plane.utils.exception_logger import log_exception from plane.utils.helpers import get_boolean_value from plane.api.serializers.issue import IssueSerializer @@ -154,6 +156,8 @@ def sanitize_issue_data(issue_data): def process_single_issue(slug, project, user_id, issue_data): try: with transaction.atomic(): + with connection.cursor() as cur: + cur.execute("SET LOCAL plane.initiator_type = 'SYSTEM.IMPORT'") # Process the main issue issue_data = sanitize_issue_data(issue_data) serializer = IssueSerializer( diff --git a/apps/api/plane/celery.py b/apps/api/plane/celery.py index 9caeeee08e..c41e119ae9 100644 --- a/apps/api/plane/celery.py +++ b/apps/api/plane/celery.py @@ -105,6 +105,11 @@ EE_JOBS = { "task": "plane.ee.bgtasks.batched_search_update_task.log_opensearch_update_queue_metrics", # noqa: E501 "schedule": crontab(minute="*/15"), # Every 15 minutes }, + # Outbox cleaner + "check-every-day-to-delete-outbox-records": { + "task": "plane.event_stream.bgtasks.outbox_cleaner.delete_outbox_records", + "schedule": crontab(hour=4, minute=0), # UTC 04:00 + }, } diff --git a/apps/api/plane/db/models/issue.py b/apps/api/plane/db/models/issue.py index e77410f4c1..6b90aab1e9 100644 --- a/apps/api/plane/db/models/issue.py +++ b/apps/api/plane/db/models/issue.py @@ -12,6 +12,9 @@ from django.db.models import Q from django import apps from django.db import connection +# Third party imports +import pgtrigger + # Module imports from plane.utils.html_processor import strip_tags from plane.db.models.project import ProjectManager diff --git a/apps/api/plane/db/mongodb.py b/apps/api/plane/db/mongodb.py deleted file mode 100644 index 9881353387..0000000000 --- a/apps/api/plane/db/mongodb.py +++ /dev/null @@ -1,25 +0,0 @@ -from pymongo import MongoClient - - -def singleton(cls): - instances = {} - - def wrapper(*args, **kwargs): - if cls not in instances: - instances[cls] = cls(*args, **kwargs) - return instances[cls] - - return wrapper - - -@singleton -class Database: - db = None - client = None - - def __init__(self, mongo_uri, database_name): - self.client = MongoClient(mongo_uri) - self.db = self.client[database_name] - - def get_db(self): - return self.db diff --git a/apps/api/plane/ee/serializers/app/epic.py b/apps/api/plane/ee/serializers/app/epic.py index 69e9417004..d40e817aff 100644 --- a/apps/api/plane/ee/serializers/app/epic.py +++ b/apps/api/plane/ee/serializers/app/epic.py @@ -2,6 +2,7 @@ from django.utils import timezone from django.core.validators import URLValidator from django.core.exceptions import ValidationError +from django.db import IntegrityError # Third Party imports from rest_framework import serializers @@ -163,54 +164,110 @@ class EpicCreateSerializer(BaseSerializer): updated_by_id = instance.updated_by_id if initiative_ids is not None: - InitiativeEpic.objects.filter(epic=instance).delete() - InitiativeEpic.objects.bulk_create( - [ - InitiativeEpic( - epic_id=instance.id, - initiative_id=initiative_id, - workspace_id=workspace_id, - created_by_id=created_by_id, - updated_by_id=updated_by_id, - ) - for initiative_id in initiative_ids - ], - batch_size=10, - ) + current_initiatives = InitiativeEpic.objects.filter( + epic=instance + ).values_list("initiative_id", flat=True) + + # Get the initiatives to add and initiatives to remove from both the current and the new initiatives + initiatives_to_add = list(set(initiative_ids) - set(current_initiatives)) + initiatives_to_remove = list(set(current_initiatives) - set(initiative_ids)) + + # Delete the initiatives to remove + InitiativeEpic.objects.filter( + epic=instance, initiative_id__in=initiatives_to_remove + ).delete() + + # Create the initiatives to add + try: + InitiativeEpic.objects.bulk_create( + [ + InitiativeEpic( + epic_id=instance.id, + initiative_id=initiative_id, + workspace_id=workspace_id, + created_by_id=created_by_id, + updated_by_id=updated_by_id, + ) + for initiative_id in initiatives_to_add + ], + batch_size=10, + ignore_conflicts=True, + ) + except IntegrityError: + pass + if assignees is not None: - IssueAssignee.objects.filter(issue=instance).delete() - IssueAssignee.objects.bulk_create( - [ - IssueAssignee( - assignee=user, - issue=instance, - project_id=project_id, - workspace_id=workspace_id, - created_by_id=created_by_id, - updated_by_id=updated_by_id, - ) - for user in assignees - ], - batch_size=10, - ) + current_assignee_ids = IssueAssignee.objects.filter( + issue=instance + ).values_list("assignee_id", flat=True) + + assignee_ids = [assignee.id for assignee in assignees] + + # Get the assignees to add and assignees to remove from both the current and the new assignees + assignees_to_add = list(set(assignee_ids) - set(current_assignee_ids)) + assignees_to_remove = list(set(current_assignee_ids) - set(assignee_ids)) + + # Delete the assignees to remove + IssueAssignee.objects.filter( + issue=instance, assignee_id__in=assignees_to_remove + ).delete() + + # Create the assignees to add + try: + IssueAssignee.objects.bulk_create( + [ + IssueAssignee( + assignee_id=user, + issue=instance, + project_id=project_id, + workspace_id=workspace_id, + created_by_id=created_by_id, + updated_by_id=updated_by_id, + ) + for user in assignees_to_add + ], + batch_size=10, + ignore_conflicts=True, + ) + except IntegrityError: + pass if labels is not None: - IssueLabel.objects.filter(issue=instance).delete() - IssueLabel.objects.bulk_create( - [ - IssueLabel( - label=label, - issue=instance, - project_id=project_id, - workspace_id=workspace_id, - created_by_id=created_by_id, - updated_by_id=updated_by_id, - ) - for label in labels - ], - batch_size=10, + current_label_ids = IssueLabel.objects.filter(issue=instance).values_list( + "label_id", flat=True ) + requested_label_ids = [label.id for label in labels] + + # Get the labels to add and labels to remove from both the current and the new labels + labels_to_add = list(set(requested_label_ids) - set(current_label_ids)) + labels_to_remove = list(set(current_label_ids) - set(requested_label_ids)) + + # Delete the labels to remove + IssueLabel.objects.filter( + issue=instance, label_id__in=labels_to_remove + ).delete() + + # Create the labels to add + try: + IssueLabel.objects.bulk_create( + [ + IssueLabel( + label_id=label, + issue=instance, + project_id=project_id, + workspace_id=workspace_id, + created_by_id=created_by_id, + updated_by_id=updated_by_id, + ) + for label in labels_to_add + ], + batch_size=10, + ignore_conflicts=True, + ) + except IntegrityError: + pass + # Time updation occues even when other related models are updated instance.updated_at = timezone.now() return super().update(instance, validated_data) diff --git a/apps/api/plane/ee/views/app/issue/bulk_operations.py b/apps/api/plane/ee/views/app/issue/bulk_operations.py index 94cde83899..ed615cbae3 100644 --- a/apps/api/plane/ee/views/app/issue/bulk_operations.py +++ b/apps/api/plane/ee/views/app/issue/bulk_operations.py @@ -6,7 +6,7 @@ from typing import List, Optional # Django imports from django.utils import timezone from django.db.models.functions import Coalesce -from django.db.models import Q, Value, UUIDField, F, Subquery, OuterRef, Prefetch +from django.db.models import Q, Value, UUIDField, Prefetch from django.contrib.postgres.fields import ArrayField from django.contrib.postgres.aggregates import ArrayAgg from django.core.serializers.json import DjangoJSONEncoder @@ -205,6 +205,10 @@ class BulkIssueOperationsEndpoint(BaseAPIView): ) for issue in issues: + # Update the updated_at and updated_by_id + issue.updated_at = timezone.now() + issue.updated_by_id = request.user.id + # Priority if properties.get("priority", False): issue_activities.append( @@ -379,9 +383,10 @@ class BulkIssueOperationsEndpoint(BaseAPIView): IssueLabel( issue=issue, label_id=label_id, - created_by=request.user, + created_by_id=request.user.id, project_id=project_id, workspace_id=workspace_id, + updated_by_id=request.user.id, ) ) bulk_issue_activities.append( @@ -411,7 +416,7 @@ class BulkIssueOperationsEndpoint(BaseAPIView): IssueAssignee( issue=issue, assignee_id=assignee_id, - created_by=request.user, + created_by_id=request.user.id, project_id=project_id, workspace_id=workspace_id, ) @@ -448,7 +453,7 @@ class BulkIssueOperationsEndpoint(BaseAPIView): module_id=module_id, project_id=project_id, workspace_id=project.workspace_id, - created_by=request.user, + created_by_id=request.user.id, ) ) issue_activities.append( @@ -549,6 +554,8 @@ class BulkIssueOperationsEndpoint(BaseAPIView): "completed_at", "estimate_point_id", "type_id", + "updated_at", + "updated_by_id", ], batch_size=100, ) diff --git a/apps/api/plane/event_stream/README.md b/apps/api/plane/event_stream/README.md new file mode 100644 index 0000000000..038ca7d67d --- /dev/null +++ b/apps/api/plane/event_stream/README.md @@ -0,0 +1,695 @@ +# πŸš€ Event Stream System + +[![Python](https://img.shields.io/badge/Python-3.12+-blue.svg)](https://python.org) +[![Django](https://img.shields.io/badge/Django-4.2+-green.svg)](https://djangoproject.com) +[![PostgreSQL](https://img.shields.io/badge/PostgreSQL-15+-blue.svg)](https://postgresql.org) +[![Async](https://img.shields.io/badge/Async-Supported-orange.svg)](https://docs.python.org/3/library/asyncio.html) + +The Event Stream system provides real-time event processing capabilities for Plane, enabling asynchronous handling of entity changes through an outbox pattern with PostgreSQL triggers and polling mechanisms. + +## πŸ“Š Performance Overview + +| Metric | Typical Value | Optimized Value | +| ---------------------- | ---------------- | ------------------ | +| **Throughput** | 1,000 events/sec | 5,000+ events/sec | +| **Latency** | < 250ms | < 100ms | +| **Memory Usage** | 200-500MB | Configurable limit | +| **CPU Usage** | 5-15% | Variable with load | +| **Batch Size** | 250 events | Tunable (50-1000) | +| **Concurrent Pollers** | 1 | Up to 10+ | + +## πŸ“‹ Table of Contents + +- [πŸ—οΈ Architecture Overview](#️-architecture-overview) +- [πŸ”§ Components](#-components) +- [βš™οΈ Configuration](#️-configuration) +- [πŸš€ Usage](#-usage) +- [πŸ’» Commands](#-commands) +- [🎯 Event Handlers](#-event-handlers) +- [🧠 Memory Management](#-memory-management) +- [πŸ‘¨β€πŸ’» Development](#-development) +- [πŸ” Troubleshooting](#-troubleshooting) +- [πŸ“ˆ Performance Tuning](#-performance-tuning) +- [πŸ“Š Monitoring & Metrics](#-monitoring--metrics) + +## πŸ—οΈ Architecture Overview + +The event stream system implements a reliable outbox pattern with the following flow: + +1. **πŸ—„οΈ Database Triggers**: PostgreSQL triggers automatically capture entity changes (INSERT/UPDATE/DELETE) and write events to the `outbox` table +2. **πŸ”„ Event Polling**: Async pollers fetch unprocessed events from the outbox table in batches +3. **⚑ Event Processing**: Registered handlers process events with full entity data +4. **βœ… Completion Tracking**: Successfully processed events are marked as `processed_at` + +### 🎯 Key Benefits + +- **πŸ”’ Reliability**: Transactional consistency ensures no events are lost +- **⚑ Performance**: Batch processing and efficient querying minimize database load +- **🧠 Memory Safety**: Built-in memory monitoring and automatic restarts prevent memory leaks +- **πŸ“ˆ Scalability**: Multiple pollers can run concurrently with competitive processing +- **πŸ”§ Flexibility**: Pluggable handler system allows custom event processing logic + +## πŸ”§ Components + +### 1. πŸ—„οΈ Models + +#### `Outbox` (`models/outbox.py`) + +Central event storage table with the following schema: + +```python +class Outbox(models.Model): + id = models.BigAutoField(primary_key=True) # Unique identifier + event_id = models.UUIDField(default=uuid4) # Event UUID + event_type = models.CharField(max_length=255) # e.g., "issue.created" + entity_type = models.CharField(max_length=255) # e.g., "issue" + entity_id = models.UUIDField() # Entity UUID + payload = models.JSONField() # Full event data + processed_at = models.DateTimeField(null=True) # Processing timestamp + created_at = models.DateTimeField(default=now) # Creation timestamp +``` + +**πŸ“Š Indexes:** + +- `outbox_unprocessed_idx`: Optimized for finding unprocessed events +- `outbox_processed_idx`: Optimized for analytics on processed events + +#### πŸ”„ Entity Proxy Models + +- `IssueProxy` (`models/issue.py`): Handles issue lifecycle events +- `CycleIssueProxy` (`models/cycle.py`): Handles cycle-issue relationship events +- Additional proxy models for other entities + +### 2. πŸš€ Outbox Poller + +#### `OutboxPoller` (`management/commands/outbox_poller.py`) + +Async poller that processes events from the outbox table: + +```python +class OutboxPoller: + def __init__(self, batch_size, interval_min, interval_max, + memory_limit_mb, memory_check_interval): + # Configurable processing parameters + # Handler registration system + # Memory monitoring +``` + +**πŸš€ Key Features:** + +- Adaptive polling intervals (backs off when no events) +- Batch processing with `FOR UPDATE SKIP LOCKED` +- Memory monitoring with automatic restarts +- Transactional safety + +#### πŸ”Œ `DatabaseConnection` (`management/commands/outbox_poller.py`) + +Async database connection manager: + +```python +class DatabaseConnection: + async def fetch_and_lock_rows(self, batch_size) -> List[tuple]: + # Fetch and lock events atomically + + async def mark_processed(self, ids: List[int]) -> bool: + # Mark events as processed +``` + +### 3. 🏭 Production Service + +#### `OutboxEventService` (`service.py`) + +Production-ready service wrapper: + +```python +class OutboxEventService: + def start(self): + # Signal handling for graceful shutdown + # Handler registration + # Service lifecycle management +``` + +## βš™οΈ Configuration + +### 🌍 Environment Variables + +```bash +# Outbox Poller Configuration +OUTBOX_POLLER_BATCH_SIZE=250 # Events per batch +OUTBOX_POLLER_INTERVAL_MIN=0.25 # Min polling interval (seconds) +OUTBOX_POLLER_INTERVAL_MAX=2.0 # Max polling interval (seconds) +OUTBOX_POLLER_MEMORY_LIMIT_MB=500 # Memory limit (MB) +OUTBOX_POLLER_MEMORY_CHECK_INTERVAL=30 # Memory check interval (seconds) +``` + +### βš™οΈ Django Settings + +Add to `INSTALLED_APPS`: + +```python +INSTALLED_APPS = [ + # ... other apps + 'plane.event_stream', +] +``` + +### πŸ—„οΈ Database Configuration + +Ensure your PostgreSQL database supports: + +- `gen_random_uuid()` function (requires `pgcrypto` extension) +- Row-level locking (`FOR UPDATE SKIP LOCKED`) + +## πŸš€ Usage + +### πŸ”„ Running the Outbox Poller + +```bash +# Run with default settings +python manage.py outbox_poller + +# Run with custom settings +python manage.py outbox_poller \ + --batch-size 100 \ + --interval-min 0.5 \ + --interval-max 5.0 \ + --memory-limit 1024 \ + --memory-check-interval 60 +``` + +### 🏭 Production Service + +```python +# service.py +from plane.event_stream.service import main + +if __name__ == "__main__": + main() +``` + +## πŸ’» Commands + +### πŸ”„ `outbox_poller` + +Async outbox poller that processes events in batches. + +**Arguments:** + +- `--batch-size`: Number of events to process per batch (default: 250) +- `--interval-min`: Minimum polling interval in seconds (default: 0.25) +- `--interval-max`: Maximum polling interval in seconds (default: 2.0) +- `--memory-limit`: Memory limit in MB before restart (default: 500) +- `--memory-check-interval`: Memory check interval in seconds (default: 30) + +**Example:** + +```bash +python manage.py outbox_poller --batch-size 500 --memory-limit 1024 +``` + +## 🎯 Event Handlers + +### πŸ“ Registering Handlers + +```python +# For outbox poller +poller = OutboxPoller(...) +poller.add_handler(my_event_handler) +``` + +### πŸ”§ Handler Function Signature + +```python +def my_event_handler(event_data: Dict[str, Any]) -> None: + """ + Process an event from the outbox. + + Args: + event_data: Dictionary containing: + - id: Event ID (int) + - event_id: Event UUID (str) + - event_type: Event type (str) + - entity_type: Entity type (str) + - entity_id: Entity UUID (str) + - payload: Event payload (dict) + - processed_at: Processing timestamp + - created_at: Creation timestamp + """ + event_type = event_data["event_type"] + payload = event_data["payload"] + + # Process the event + if event_type == "issue.created": + handle_new_issue(payload) + elif event_type == "issue.updated": + handle_issue_update(payload) +``` + +### ⚑ Async Handlers + +Both sync and async handlers are supported: + +```python +async def async_event_handler(event_data: Dict[str, Any]) -> None: + """Async event handler for non-blocking processing.""" + await send_webhook(event_data) + await update_search_index(event_data) +``` + +### πŸ“ Example Handlers + +#### πŸ†• Issue Created Handler + +```python +def handle_issue_created(event_data: Dict[str, Any]): + payload = event_data["payload"] + issue_data = payload["data"] + + # Send webhook notification + send_webhook("issue_created", issue_data) + + # Update search index + update_search_index("issue", issue_data) + + # Send notifications to assignees + notify_assignees(issue_data.get("assignees", [])) +``` + +#### πŸ”„ Issue Updated Handler + +```python +def handle_issue_updated(event_data: Dict[str, Any]): + payload = event_data["payload"] + issue_data = payload["data"] + previous_data = payload.get("previous_attributes", {}) + + # Check what changed + if "state_id" in previous_data: + handle_state_change(issue_data, previous_data) + + if "assignees" in previous_data: + handle_assignee_change(issue_data, previous_data) +``` + +## 🧠 Memory Management + +The event stream system includes comprehensive memory management to prevent memory leaks and ensure long-running stability. + +### πŸ“Š Memory Monitoring + +The outbox poller monitors memory usage: + +```python +class MemoryMonitor: + def __init__(self, memory_limit_mb: int, check_interval: int): + # Track memory usage with psutil + # Automatic restart when limits exceeded +``` + +### πŸ”„ Automatic Restarts + +When memory limits are exceeded: + +1. Current processing completes +2. Resources are cleaned up +3. Process restarts with fresh memory +4. Event processing resumes + +### πŸ’‘ Best Practices + +1. **Set appropriate memory limits** based on your environment +2. **Monitor restart frequency** - frequent restarts may indicate issues +3. **Use batch processing** to minimize memory accumulation +4. **Clean up resources** in event handlers + +## πŸ‘¨β€πŸ’» Development + +### πŸ› οΈ Setting Up Development Environment + +1. **Install dependencies**: + +```bash +pip install psycopg[async] psutil +``` + +2. **Run migrations**: + +```bash +python manage.py migrate event_stream +``` + +3. **Set up database triggers**: + +```bash +python manage.py migrate +``` + +### πŸ§ͺ Testing Event Generation + +Create test events manually: + +```python +from plane.event_stream.models import Outbox + +# Create a test event +Outbox.objects.create( + event_type="test.event", + entity_type="test", + entity_id="123e4567-e89b-12d3-a456-426614174000", + payload={"message": "Hello, World!"} +) +``` + +### πŸ†• Custom Event Types + +Add new event types by: + +1. **Creating proxy models** with appropriate triggers +2. **Registering event handlers** for the new event types +3. **Testing event generation** and processing + +## πŸ” Troubleshooting + +### ❌ Common Issues + +#### 🚫 No Events Being Processed + +**Symptoms**: Poller runs but no events are processed + +**Diagnosis**: + +```sql +-- Check for unprocessed events +SELECT COUNT(*) FROM outbox WHERE processed_at IS NULL; + +-- Check recent events +SELECT * FROM outbox ORDER BY created_at DESC LIMIT 10; +``` + +**Solutions**: + +- Verify database triggers are installed +- Check handler registration +- Review error logs for handler exceptions + +#### 🧠 Memory Restarts Too Frequent + +**Symptoms**: Process restarts every few minutes + +**Diagnosis**: + +- Check memory limit settings +- Monitor actual memory usage +- Review handler efficiency + +**Solutions**: + +- Increase memory limits +- Optimize handler memory usage +- Reduce batch sizes +- Check for memory leaks in handlers + +#### πŸ”Œ Database Connection Issues + +**Symptoms**: Connection errors or timeouts + +**Solutions**: + +- Verify database settings +- Check connection pool configuration +- Review PostgreSQL logs +- Ensure `pgcrypto` extension is installed + +#### πŸ”„ Events Being Processed Multiple Times + +**Symptoms**: Duplicate event processing + +**Diagnosis**: + +- Check database row locking +- Verify transaction boundaries + +**Solutions**: + +- Ensure proper use of `FOR UPDATE SKIP LOCKED` +- Check for handler exceptions +- Review transaction boundaries + +## πŸ“ˆ Performance Tuning + +### πŸš€ Throughput Optimization + +#### Batch Size Tuning + +```bash +# Low latency (real-time processing) +python manage.py outbox_poller --batch-size 50 --interval-min 0.1 + +# High throughput (batch processing) +python manage.py outbox_poller --batch-size 1000 --interval-min 1.0 + +# Balanced (recommended for most use cases) +python manage.py outbox_poller --batch-size 250 --interval-min 0.25 +``` + +#### Concurrent Processing + +```bash +# Run multiple pollers for horizontal scaling +# Terminal 1 +python manage.py outbox_poller --batch-size 200 + +# Terminal 2 +python manage.py outbox_poller --batch-size 200 + +# Terminal 3 +python manage.py outbox_poller --batch-size 200 +``` + +### 🎯 Performance Benchmarks + +| Configuration | Events/sec | Memory (MB) | CPU (%) | Latency (ms) | +| ------------------------------ | ---------- | ----------- | ------- | ------------ | +| **Single Poller (batch=50)** | 500 | 150 | 8 | 100 | +| **Single Poller (batch=250)** | 1,200 | 200 | 12 | 200 | +| **Single Poller (batch=1000)** | 2,500 | 350 | 18 | 400 | +| **3 Pollers (batch=250 each)** | 3,000 | 600 | 30 | 250 | +| **5 Pollers (batch=200 each)** | 4,500 | 800 | 45 | 300 | + +### ⚑ Database Optimization + +#### Index Optimization + +```sql +-- Monitor index usage +SELECT + schemaname, + tablename, + indexname, + idx_scan, + idx_tup_read, + idx_tup_fetch +FROM pg_stat_user_indexes +WHERE tablename = 'outbox' +ORDER BY idx_scan DESC; + +-- Monitor table statistics +SELECT + seq_scan, + seq_tup_read, + idx_scan, + idx_tup_fetch, + n_tup_ins, + n_tup_upd, + n_tup_del +FROM pg_stat_user_tables +WHERE relname = 'outbox'; +``` + +#### Table Maintenance + +```sql +-- Regular cleanup of processed events (run daily) +DELETE FROM outbox +WHERE processed_at < NOW() - INTERVAL '7 days'; + +-- Vacuum and analyze for performance +VACUUM ANALYZE outbox; + +-- Monitor table size +SELECT + pg_size_pretty(pg_total_relation_size('outbox')) as total_size, + pg_size_pretty(pg_relation_size('outbox')) as table_size, + pg_size_pretty(pg_indexes_size('outbox')) as index_size; +``` + +## πŸ“Š Monitoring & Metrics + +### πŸ“ˆ Key Performance Indicators (KPIs) + +#### Real-time Metrics + +```python +# Add to your monitoring system +METRICS = { + 'event_processing_rate': 'events_per_second', + 'queue_depth': 'unprocessed_events_count', + 'memory_usage': 'memory_mb', + 'cpu_usage': 'cpu_percentage', + 'error_rate': 'errors_per_minute', + 'restart_frequency': 'restarts_per_hour', + 'average_latency': 'processing_time_ms' +} +``` + +#### Health Check Endpoint + +```python +# Add to your Django views +def event_stream_health(request): + unprocessed_count = Outbox.objects.filter(processed_at__isnull=True).count() + last_processed = Outbox.objects.filter( + processed_at__isnull=False + ).order_by('-processed_at').first() + + health_status = { + 'status': 'healthy' if unprocessed_count < 1000 else 'warning', + 'unprocessed_events': unprocessed_count, + 'last_processed_at': last_processed.processed_at if last_processed else None, + 'queue_health': 'ok' if unprocessed_count < 5000 else 'critical' + } + + return JsonResponse(health_status) +``` + +### πŸ“Š Monitoring + +#### πŸ“Š Key Metrics to Monitor + +1. **⚑ Event Processing Rate**: Events processed per second +2. **🧠 Memory Usage**: Current and peak memory consumption +3. **πŸ”„ Restart Frequency**: How often processes restart +4. **❌ Error Rate**: Handler failure percentage +5. **πŸ“¦ Queue Depth**: Number of unprocessed events +6. **⏱️ Processing Latency**: Time from event creation to processing +7. **πŸ”’ Lock Contention**: Database lock wait times + +#### πŸ“Š Database Queries for Monitoring + +```sql +-- Unprocessed events by type +SELECT event_type, COUNT(*) +FROM outbox +WHERE processed_at IS NULL +GROUP BY event_type; + +-- Processing rate (last hour) +SELECT DATE_TRUNC('minute', processed_at) as minute, + COUNT(*) as processed_count +FROM outbox +WHERE processed_at > NOW() - INTERVAL '1 hour' +GROUP BY minute +ORDER BY minute; + +-- Average processing time +SELECT event_type, + AVG(EXTRACT(EPOCH FROM (processed_at - created_at))) as avg_seconds +FROM outbox +WHERE processed_at IS NOT NULL +GROUP BY event_type; + +-- Queue depth over time +SELECT + DATE_TRUNC('hour', created_at) as hour, + COUNT(*) as events_created, + COUNT(processed_at) as events_processed, + COUNT(*) - COUNT(processed_at) as queue_depth +FROM outbox +WHERE created_at > NOW() - INTERVAL '24 hours' +GROUP BY hour +ORDER BY hour; + +-- Error rate monitoring +SELECT + DATE_TRUNC('hour', created_at) as hour, + COUNT(*) as total_events, + COUNT(CASE WHEN processed_at IS NULL AND created_at < NOW() - INTERVAL '5 minutes' THEN 1 END) as failed_events, + ROUND( + (COUNT(CASE WHEN processed_at IS NULL AND created_at < NOW() - INTERVAL '5 minutes' THEN 1 END) * 100.0) / COUNT(*), + 2 + ) as error_rate_percent +FROM outbox +WHERE created_at > NOW() - INTERVAL '24 hours' +GROUP BY hour +ORDER BY hour; +``` + +#### πŸ”” Alerting Thresholds + +| Metric | Warning | Critical | Action | +| --------------------- | ----------- | ----------- | -------------------- | +| **Queue Depth** | > 1,000 | > 5,000 | Scale pollers | +| **Processing Rate** | < 100/sec | < 50/sec | Check handlers | +| **Error Rate** | > 5% | > 15% | Investigation needed | +| **Memory Usage** | > 80% limit | > 95% limit | Restart/scale | +| **Restart Frequency** | > 5/hour | > 20/hour | Check memory leaks | + +### πŸ› οΈ Logging + +The system uses structured logging with the following loggers: + +- `plane.event_stream`: Main event stream operations +- `plane.event_stream.poller`: Outbox poller specific logs + +Configure logging levels as needed: + +```python +LOGGING = { + 'loggers': { + 'plane.event_stream': { + 'level': 'INFO', + 'handlers': ['console', 'file'], + }, + }, +} +``` + +## Performance Considerations + +### πŸ—„οΈ Database Optimization + +1. **Indexes**: Ensure proper indexes on the outbox table +2. **Partitioning**: Consider partitioning for high-volume scenarios +3. **Cleanup**: Regularly clean up old processed events + +### πŸ“ˆ Scaling + +1. **πŸ”„ Multiple Pollers**: Run multiple poller instances for higher throughput +2. **πŸ“¦ Batch Sizes**: Tune batch sizes based on memory and latency requirements +3. **⚑ Handler Optimization**: Optimize handlers for minimal memory usage and fast processing + +### πŸ’‘ Best Practices + +1. **πŸ” Idempotent Handlers**: Ensure handlers can be safely retried +2. **❌ Error Handling**: Implement proper error handling and logging +3. **🧹 Resource Cleanup**: Always clean up resources in handlers +4. **πŸ“Š Monitoring**: Implement comprehensive monitoring and alerting + +--- + +## πŸ“š Additional Resources + +- πŸ“– [Event Stream Specification](../../../EVENT_STREAM_SPECIFICATION.md) - Detailed event schemas and processing requirements +- 🐘 [PostgreSQL Triggers Documentation](https://www.postgresql.org/docs/current/sql-createtrigger.html) - Official PostgreSQL trigger documentation +- ⚑ [Async Python Guide](https://docs.python.org/3/library/asyncio.html) - Python asyncio documentation +- πŸ”§ [psycopg3 Documentation](https://www.psycopg.org/psycopg3/docs/) - Async PostgreSQL adapter + +## 🀝 Contributing + +When contributing to the event stream system: + +1. **πŸ“ Add tests** for new event types and handlers +2. **πŸ“Š Include performance benchmarks** for significant changes +3. **πŸ“– Update documentation** including this README +4. **πŸ” Test memory usage** with realistic workloads +5. **⚑ Benchmark throughput** before and after changes diff --git a/apps/api/plane/event_stream/__init__.py b/apps/api/plane/event_stream/__init__.py new file mode 100644 index 0000000000..8b13789179 --- /dev/null +++ b/apps/api/plane/event_stream/__init__.py @@ -0,0 +1 @@ + diff --git a/apps/api/plane/event_stream/apps.py b/apps/api/plane/event_stream/apps.py new file mode 100644 index 0000000000..61a106ad74 --- /dev/null +++ b/apps/api/plane/event_stream/apps.py @@ -0,0 +1,5 @@ +from django.apps import AppConfig + + +class EventStreamConfig(AppConfig): + name = "plane.event_stream" diff --git a/apps/api/plane/event_stream/bgtasks/__init__.py b/apps/api/plane/event_stream/bgtasks/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/apps/api/plane/event_stream/bgtasks/outbox_cleaner.py b/apps/api/plane/event_stream/bgtasks/outbox_cleaner.py new file mode 100644 index 0000000000..6f27a3b80d --- /dev/null +++ b/apps/api/plane/event_stream/bgtasks/outbox_cleaner.py @@ -0,0 +1,189 @@ +# Python imports +import os +import logging +from datetime import timedelta +from typing import List, Dict, Any + +# Django imports +from django.utils import timezone + +# Third party imports +from celery import shared_task +from pymongo.errors import BulkWriteError +from pymongo import collection +from pymongo.operations import InsertOne + +# Module imports +from plane.event_stream.models.outbox import Outbox +from plane.utils.exception_logger import log_exception +from plane.settings.mongo import MongoConnection + + +BATCH_SIZE: int = int(os.environ.get("OUTBOX_CLEANER_BATCH_SIZE", 1000)) + + +# Set up logger +logger = logging.getLogger("plane.worker") + + +def flush_to_mongo_and_delete( + mongo_collection: collection, buffer: List[Dict[str, Any]], ids_to_delete: List[int] +) -> None: + """ + Inserts a batch of API logs into MongoDB and deletes the corresponding rows from PostgreSQL. + + Args: + mongo_collection (Collection): The MongoDB collection to insert documents into. + buffer (List[Dict[str, Any]]): List of API log documents to insert into MongoDB. + ids_to_delete (List[int]): List of primary key IDs of logs to delete from PostgreSQL. + """ + if not buffer: + logger.debug("No records to flush - buffer is empty") + return + + logger.info( + f"Starting batch flush: {len(buffer)} records, {len(ids_to_delete)} IDs to delete" + ) + + try: + # Insert into MongoDB + mongo_collection.bulk_write([InsertOne(doc) for doc in buffer]) + except BulkWriteError as bwe: + logger.error(f"MongoDB bulk write error: {str(bwe)}") + log_exception(bwe) + # Continue with deletion even if MongoDB insert fails + + # Delete from PostgreSQL + deleted_count = Outbox.objects.filter(id__in=ids_to_delete).delete()[0] + + logger.info( + "Batch flush completed", + extra={ + "batch_size": deleted_count, + }, + ) + + +@shared_task +def delete_outbox_records() -> None: + """ + Celery background task that migrates outbox records older than 2 days + from PostgreSQL to MongoDB, then deletes them from PostgreSQL. + + - Reads outbox records in chunks to reduce memory usage. + - Writes to MongoDB using unordered bulk inserts for efficiency. + - Deletes processed outbox records in batches to minimize write pressure. + """ + + logger.info("Starting outbox cleanup task") + + # Check MongoDB availability + mongo_available = MongoConnection.is_configured() + logger.info(f"MongoDB configured: {mongo_available}") + + mongo_collection = None + if mongo_available: + try: + collection_name = os.environ.get("OUTBOX_COLLECTION_NAME", "outbox") + mongo_collection = MongoConnection.get_collection(collection_name) + logger.info( + f"MongoDB collection '{collection_name}' connected successfully" + ) + except Exception as e: + logger.error(f"Failed to get MongoDB collection: {str(e)}") + log_exception(e) + mongo_available = False + + # Calculate cutoff time + cutoff_days = int(os.environ.get("OUTBOX_CLEANER_CUTOFF_DAYS", 2)) + cutoff_time = timezone.now() - timedelta(days=cutoff_days) + logger.info( + f"Processing outbox records older than {cutoff_time} (cutoff: {cutoff_days} days)" + ) + + # Get records to process + queryset = ( + Outbox.objects.filter(processed_at__lte=cutoff_time) + .values( + "id", + "event_id", + "event_type", + "entity_type", + "entity_id", + "payload", + "processed_at", + "created_at", + "workspace_id", + "project_id", + ) + .iterator(chunk_size=BATCH_SIZE) + ) + + buffer: List[Dict[str, Any]] = [] + ids_to_delete: List[int] = [] + total_processed = 0 + total_batches = 0 + + logger.info(f"Starting to process outbox records with batch size: {BATCH_SIZE}") + + for log in queryset: + buffer.append( + { + "event_id": log["event_id"], + "event_type": log["event_type"], + "entity_type": log["entity_type"], + "entity_id": log["entity_id"], + "payload": log["payload"], + "processed_at": log["processed_at"], + "created_at": log["created_at"], + "workspace_id": log["workspace_id"], + "project_id": log["project_id"], + } + ) + ids_to_delete.append(log["id"]) + + if len(buffer) >= BATCH_SIZE: + total_batches += 1 + logger.debug(f"Processing batch {total_batches}: {len(buffer)} records") + + if mongo_available: + flush_to_mongo_and_delete(mongo_collection, buffer, ids_to_delete) + else: + deleted_count = Outbox.objects.filter(id__in=ids_to_delete).delete()[0] + logger.info( + f"Deleted {deleted_count} records from PostgreSQL (MongoDB unavailable)" + ) + + total_processed += len(buffer) + + # Clear the buffer and ids_to_delete + buffer.clear() + ids_to_delete.clear() + + # Flush remaining outbox records + if buffer: + total_batches += 1 + logger.debug(f"Processing final batch {total_batches}: {len(buffer)} records") + + if mongo_available: + flush_to_mongo_and_delete(mongo_collection, buffer, ids_to_delete) + else: + deleted_count = Outbox.objects.filter(id__in=ids_to_delete).delete()[0] + logger.info( + f"Deleted {deleted_count} records from PostgreSQL (MongoDB unavailable)" + ) + + total_processed += len(buffer) + + # Final summary log + logger.info( + "Outbox cleanup task completed", + extra={ + "total_records_processed": total_processed, + "total_batches": total_batches, + "mongo_available": mongo_available, + "cutoff_time": cutoff_time, + }, + ) + + return diff --git a/apps/api/plane/event_stream/management/__init__.py b/apps/api/plane/event_stream/management/__init__.py new file mode 100644 index 0000000000..b784d7ddd2 --- /dev/null +++ b/apps/api/plane/event_stream/management/__init__.py @@ -0,0 +1 @@ +# Management package for event_publisher app diff --git a/apps/api/plane/event_stream/management/commands/__init__.py b/apps/api/plane/event_stream/management/commands/__init__.py new file mode 100644 index 0000000000..249652568c --- /dev/null +++ b/apps/api/plane/event_stream/management/commands/__init__.py @@ -0,0 +1 @@ +# Management commands package for event_publisher app diff --git a/apps/api/plane/event_stream/management/commands/outbox_poller.py b/apps/api/plane/event_stream/management/commands/outbox_poller.py new file mode 100644 index 0000000000..981e2fc52f --- /dev/null +++ b/apps/api/plane/event_stream/management/commands/outbox_poller.py @@ -0,0 +1,789 @@ +# Standard Library imports +import asyncio +import logging +import os +import signal +import sys +from typing import List, Callable, Dict, Any +from urllib.parse import quote + +# Third Party imports +import psycopg +import psutil +import psycopg_pool + +# Django imports +from django.core.management.base import BaseCommand +from django.conf import settings +from django.utils import timezone + +# Module imports +from plane.utils.exception_logger import log_exception +from plane.event_stream.models.outbox import OutboxEvent + + +# Defaults +BATCH_SIZE = int(os.environ.get("OUTBOX_POLLER_BATCH_SIZE", 250)) +INTERVAL_MIN = float(os.environ.get("OUTBOX_POLLER_INTERVAL_MIN", 0.25)) +INTERVAL_MAX = float(os.environ.get("OUTBOX_POLLER_INTERVAL_MAX", 2.0)) +MEMORY_LIMIT_MB = int(os.environ.get("OUTBOX_POLLER_MEMORY_LIMIT_MB", 500)) +MEMORY_CHECK_INTERVAL = int(os.environ.get("OUTBOX_POLLER_MEMORY_CHECK_INTERVAL", 30)) + +# Connection Pool Configuration +POOL_SIZE = int(os.environ.get("OUTBOX_POLLER_POOL_SIZE", 4)) +POOL_MIN_SIZE = int(os.environ.get("OUTBOX_POLLER_POOL_MIN_SIZE", 2)) +POOL_MAX_SIZE = int(os.environ.get("OUTBOX_POLLER_POOL_MAX_SIZE", 10)) +POOL_TIMEOUT = float(os.environ.get("OUTBOX_POLLER_POOL_TIMEOUT", 30.0)) +POOL_MAX_IDLE = float(os.environ.get("OUTBOX_POLLER_POOL_MAX_IDLE", 300.0)) +POOL_MAX_LIFETIME = float(os.environ.get("OUTBOX_POLLER_POOL_MAX_LIFETIME", 3600.0)) +POOL_RECONNECT_TIMEOUT = float( + os.environ.get("OUTBOX_POLLER_POOL_RECONNECT_TIMEOUT", 5.0) +) +POOL_HEALTH_CHECK_INTERVAL = int( + os.environ.get("OUTBOX_POLLER_POOL_HEALTH_CHECK_INTERVAL", 60) +) + +# Configuration +RESTART_EXIT_CODE = 100 +GRACEFUL_SHUTDOWN_TIMEOUT = 30 # seconds + +log = logging.getLogger("plane.event_stream") + + +class GracefulShutdownHandler: + """ + Handles graceful shutdown signals for the outbox poller. + """ + + def __init__(self): + self.shutdown_event = asyncio.Event() + self.signal_received = None + + def setup_signal_handlers(self): + """Set up signal handlers for graceful shutdown.""" + # Only set up signal handlers if running in the main thread + try: + # Handle SIGTERM (sent by Kubernetes when downsizing pods) + signal.signal(signal.SIGTERM, self._signal_handler) + # Handle SIGINT (Ctrl+C) + signal.signal(signal.SIGINT, self._signal_handler) + # Handle SIGQUIT (Ctrl+\) + signal.signal(signal.SIGQUIT, self._signal_handler) + + log.info("Signal handlers registered for SIGTERM, SIGINT, SIGQUIT") + except ValueError as e: + # This can happen if not running in main thread + log.warning(f"Could not register signal handlers: {e}") + + def _signal_handler(self, signum, frame): + """Handle shutdown signals.""" + signal_names = { + signal.SIGTERM: "SIGTERM", + signal.SIGINT: "SIGINT", + signal.SIGQUIT: "SIGQUIT", + } + + signal_name = signal_names.get(signum, f"Signal {signum}") + self.signal_received = signal_name + + log.info(f"Received {signal_name}, initiating graceful shutdown...") + + # Set the shutdown event to signal all async tasks + if not self.shutdown_event.is_set(): + self.shutdown_event.set() + + def shutdown_requested(self) -> bool: + """Check if shutdown has been requested.""" + return self.shutdown_event.is_set() + + async def wait_for_shutdown(self, timeout: float = None): + """Wait for shutdown signal with optional timeout.""" + if timeout: + try: + await asyncio.wait_for(self.shutdown_event.wait(), timeout=timeout) + except asyncio.TimeoutError: + pass + else: + await self.shutdown_event.wait() + + +class MemoryMonitor: + """ + Async memory monitor that checks memory usage periodically and signals restart if limit exceeded. + """ + + def __init__(self, memory_limit_mb: int, check_interval: int): + self.memory_limit_mb = memory_limit_mb + self.check_interval = check_interval + self.process = psutil.Process(os.getpid()) + self._running = False + self._restart_event = asyncio.Event() # Use asyncio.Event for signaling + + async def start(self): + """Start the memory monitoring loop.""" + self._running = True + while self._running: + await asyncio.sleep(self.check_interval) + await self._check_memory() + + async def stop(self): + """Stop the memory monitoring loop.""" + self._running = False + + def restart_requested(self) -> bool: + """Check if restart has been requested.""" + return self._restart_event.is_set() + + async def wait_for_restart(self): + """Wait for restart signal.""" + await self._restart_event.wait() + + async def _check_memory(self): + """Check current memory usage and request restart if limit exceeded.""" + try: + mem_mb = self.process.memory_info().rss / 1024 / 1024 + log.info( + "Memory usage: %.2f MB", + mem_mb, + extra={"memory_usage": round(mem_mb, 2)}, + ) + + if mem_mb > self.memory_limit_mb: + log.warning( + "Memory usage exceeded limit - requesting restart", + extra={ + "memory_usage": round(mem_mb, 2), + "memory_limit": self.memory_limit_mb, + }, + ) + self._restart_event.set() # Signal restart + self._running = False # Stop the monitoring loop + except Exception as e: + log_exception(e) + log.warning(f"Failed to check memory usage: {e}") + + +class DatabaseConnectionPool: + """ + Manages a psycopg3 async connection pool for outbox polling. + Includes health checking, connection lifecycle management, and automatic reconnection. + """ + + def __init__( + self, + pool_size: int = POOL_SIZE, + min_size: int = POOL_MIN_SIZE, + max_size: int = POOL_MAX_SIZE, + timeout: float = POOL_TIMEOUT, + max_idle: float = POOL_MAX_IDLE, + max_lifetime: float = POOL_MAX_LIFETIME, + reconnect_timeout: float = POOL_RECONNECT_TIMEOUT, + health_check_interval: int = POOL_HEALTH_CHECK_INTERVAL, + ): + self.pool: psycopg_pool.AsyncConnectionPool | None = None + self.dsn = self._get_dsn_from_settings() + self.pool_size = pool_size + self.min_size = min_size + self.max_size = max_size + self.timeout = timeout + self.max_idle = max_idle + self.max_lifetime = max_lifetime + self.reconnect_timeout = reconnect_timeout + self.health_check_interval = health_check_interval + self._health_check_task: asyncio.Task | None = None + self._running = False + + def _get_dsn_from_settings(self) -> str: + """Extract and normalize the DSN from Django settings.""" + dsn = getattr(settings, "DATABASE_URL", None) + + # If DATABASE_URL is not set, use the default database settings + if not dsn: + db = settings.DATABASES["default"] + dsn = ( + f"postgresql://{quote(db['USER'])}:{quote(db['PASSWORD'])}" + f"@{db['HOST']}:{db['PORT']}/{db['NAME']}" + ) + + # If the DSN starts with postgres://, replace it with postgresql:// + if dsn.startswith("postgres://"): + dsn = dsn.replace("postgres://", "postgresql://", 1) + return dsn + + async def connect(self): + """Initialize the connection pool with comprehensive configuration.""" + try: + # Create connection pool with advanced configuration + self.pool = psycopg_pool.AsyncConnectionPool( + conninfo=self.dsn, + min_size=self.min_size, + max_size=self.max_size, + timeout=self.timeout, + max_idle=self.max_idle, + max_lifetime=self.max_lifetime, + reconnect_timeout=self.reconnect_timeout, + # Configure connection preparation + configure=self._configure_connection, + # Reset connections on return to pool + reset=self._reset_connection, + # Explicitly set open=False to avoid deprecation warning + open=False, + ) + + # Open the pool explicitly + await self.pool.open() + + # Test the pool with a simple query + async with self.pool.connection() as conn: + await conn.execute("SELECT 1") + + log.info( + "Connection pool established successfully", + extra={ + "pool_size": self.pool_size, + "min_size": self.min_size, + "max_size": self.max_size, + "timeout": self.timeout, + "max_idle": self.max_idle, + "max_lifetime": self.max_lifetime, + }, + ) + + # Start health check monitoring + self._running = True + self._health_check_task = asyncio.create_task(self._health_check_loop()) + + except Exception as e: + log.exception(f"Failed to establish connection pool: {e}") + await self.close() + raise + + async def _configure_connection(self, conn: psycopg.AsyncConnection): + """Configure individual connections when they're created.""" + # Set autocommit to False for explicit transaction control + await conn.set_autocommit(False) + + async def _reset_connection(self, conn: psycopg.AsyncConnection): + """Reset connection state when returned to pool.""" + # Rollback any uncommitted transactions + try: + await conn.rollback() + except Exception: + # Connection might already be in a good state + pass + + async def _health_check_loop(self): + """Periodic health check for the connection pool.""" + while self._running: + try: + await asyncio.sleep(self.health_check_interval) + if self._running: + health_status = await self.health_check() + if not health_status["healthy"]: + log.warning( + "Connection pool health check failed", + extra=health_status, + ) + except Exception as e: + log.exception(f"Error in health check loop: {e}") + + async def health_check(self) -> Dict[str, Any]: + """ + Comprehensive health check of the connection pool. + Returns detailed health status information. + """ + if not self.pool: + return { + "healthy": False, + "error": "Pool not initialized", + "timestamp": timezone.now().isoformat(), + } + + try: + # Get pool statistics + pool_stats = self.pool.get_stats() + + stats = { + "pool_size": pool_stats.get("pool_size", 0), + "available_connections": pool_stats.get("pool_available", 0), + "used_connections": pool_stats.get("pool_size", 0) + - pool_stats.get("pool_available", 0), + "waiting_requests": pool_stats.get("requests_waiting", 0), + "timestamp": timezone.now().isoformat(), + } + + # Test connection with a simple query + start_time = asyncio.get_event_loop().time() + async with self.pool.connection() as conn: + await conn.execute("SELECT 1") + response_time = (asyncio.get_event_loop().time() - start_time) * 1000 + + stats.update( + { + "healthy": True, + "response_time_ms": round(response_time, 2), + "pool_status": "operational", + } + ) + + log.info("Pool health check passed", extra=stats) + return stats + + except Exception as e: + error_stats = { + "healthy": False, + "error": str(e), + "error_type": type(e).__name__, + "timestamp": timezone.now().isoformat(), + } + + if self.pool: + try: + pool_stats = self.pool.get_stats() + error_stats.update( + { + "pool_size": pool_stats.get("pool_size", 0), + "available_connections": pool_stats.get( + "pool_available", 0 + ), + } + ) + except Exception: + # If we can't get stats, that's additional confirmation something is wrong + pass + + log.error("Pool health check failed", extra=error_stats) + return error_stats + + async def get_pool_stats(self) -> Dict[str, Any]: + """Get current pool statistics.""" + if not self.pool: + return {"error": "Pool not initialized"} + + try: + stats = self.pool.get_stats() + + # Add configuration info to the stats + stats.update( + { + "min_size": self.min_size, + "max_size": self.max_size, + "timeout": self.timeout, + "max_idle": self.max_idle, + "max_lifetime": self.max_lifetime, + "reconnect_timeout": self.reconnect_timeout, + } + ) + + return stats + except Exception as e: + return {"error": f"Failed to get pool stats: {str(e)}"} + + async def fetch_and_lock_rows(self, batch_size: int) -> List[tuple]: + """Fetch and lock rows from outbox table using FOR UPDATE SKIP LOCKED.""" + if not self.pool: + return [] + + try: + async with self.pool.connection() as conn: + async with conn.transaction(): + async with conn.cursor() as cur: + await cur.execute( + """ + UPDATE outbox + SET claimed_at = NOW() + WHERE id IN ( + SELECT id FROM outbox + WHERE processed_at IS NULL + AND claimed_at IS NULL + ORDER BY id + LIMIT %s + FOR UPDATE SKIP LOCKED + ) + RETURNING id, event_id, event_type, entity_type, entity_id, + payload, processed_at, created_at, claimed_at, + workspace_id, project_id, initiator_id, initiator_type; + """, + (batch_size,), + ) + return await cur.fetchall() + except Exception as e: + log.error(f"Error fetching and locking rows: {e}") + return [] + + async def mark_processed(self, ids: List[int]) -> bool: + """Mark events as processed.""" + if not self.pool or not ids: + return False + + try: + async with self.pool.connection() as conn: + async with conn.transaction(): + async with conn.cursor() as cur: + await cur.execute( + """ + UPDATE outbox + SET processed_at = NOW() + WHERE id = ANY(%s); + """, + (ids,), + ) + rows_updated = cur.rowcount + log.info(f"Marked {rows_updated} rows as processed") + return rows_updated > 0 + except Exception as e: + log.error(f"Error marking events as processed: {e}") + return False + + async def close(self): + """Close the connection pool and cleanup resources.""" + self._running = False + + # Cancel health check task + if self._health_check_task: + self._health_check_task.cancel() + try: + await self._health_check_task + except asyncio.CancelledError: + pass + self._health_check_task = None + + # Close the pool + if self.pool: + try: + await self.pool.close() + log.info("Connection pool closed successfully") + except Exception as e: + log.warning(f"Error closing connection pool: {e}") + self.pool = None + + async def __aenter__(self) -> "DatabaseConnectionPool": + await self.connect() + return self + + async def __aexit__(self, exc_type, exc_value, traceback): + await self.close() + + +class OutboxPoller: + """ + Async outbox poller with event handler registration pattern and connection pooling. + """ + + def __init__( + self, + batch_size: int, + interval_min: float, + interval_max: float, + memory_limit_mb: int, + memory_check_interval: int, + ): + self.batch_size = batch_size + self.interval_min = interval_min + self.interval_max = interval_max + self.memory_limit_mb = memory_limit_mb + self.memory_check_interval = memory_check_interval + self.handlers: List[Callable[[Dict[str, Any]], None]] = [] + self.shutdown_handler = GracefulShutdownHandler() + + def add_handler(self, handler: Callable[[Dict[str, Any]], None]) -> None: + """ + Add a handler function to be invoked with each event. + :param handler: Callable accepting event data dict + """ + self.handlers.append(handler) + log.info(f"Handler registered: {handler.__name__}") + + async def _process_event(self, row: tuple) -> bool: + """ + Process a single event by calling all registered handlers. + Returns True if all handlers succeeded, False otherwise. + """ + # Unpack all fields from the row + event = OutboxEvent.from_db_row(row) + + handler_errors = [] + for handler in self.handlers: + try: + # Check if handler is async + if asyncio.iscoroutinefunction(handler): + await handler(event) + else: + handler(event) + except Exception as e: + handler_errors.append((handler.__name__, str(e))) + log.exception(f"Error in handler '{handler.__name__}': {e}") + + if handler_errors: + log.error( + f"Event {event.event_id} had {len(handler_errors)} handler errors: {handler_errors}" + ) + return False + + return True + + async def start(self): + """Start the polling loop with connection pooling and signal handling.""" + # Set up signal handlers + self.shutdown_handler.setup_signal_handlers() + + delay = self.interval_min + empty_cycles = 0 + + # Start memory monitoring + memory_monitor = MemoryMonitor(self.memory_limit_mb, self.memory_check_interval) + memory_task = asyncio.create_task(memory_monitor.start()) + + try: + # Initialize database connection pool + async with DatabaseConnectionPool() as db_pool: + + # Log initial pool health + health_status = await db_pool.health_check() + log.info("Initial pool health check", extra=health_status) + + while True: + # Check for shutdown signal first + if self.shutdown_handler.shutdown_requested(): + log.info( + f"Shutdown signal received ({self.shutdown_handler.signal_received}), stopping poller..." + ) + break + + # Check for memory limit exceeded + if memory_monitor.restart_requested(): + log.warning("Memory limit exceeded - initiating restart") + break + + rows = await db_pool.fetch_and_lock_rows(self.batch_size) + if not rows: + log.info("No rows to process. Sleeping for %s seconds.", delay) + empty_cycles += 1 + # Increase delay every 5 consecutive empty cycles + if empty_cycles % 5 == 0: + new_delay = min(delay * 2, self.interval_max) + log.info( + "No rows to process for %d cycles. Increasing delay from %s to %s seconds.", + empty_cycles, + delay, + new_delay, + ) + delay = new_delay + + # Use asyncio.wait_for with timeout to check for shutdown or restart + shutdown_tasks = [ + asyncio.create_task( + self.shutdown_handler.wait_for_shutdown() + ), + asyncio.create_task(memory_monitor.wait_for_restart()), + ] + + try: + done, pending = await asyncio.wait( + shutdown_tasks, + timeout=delay, + return_when=asyncio.FIRST_COMPLETED, + ) + + # Cancel any pending tasks + for task in pending: + task.cancel() + try: + await task + except asyncio.CancelledError: + pass + + # Check what completed + if done: + # Either shutdown or restart was requested + if self.shutdown_handler.shutdown_requested(): + log.info("Shutdown signal received during sleep") + break + elif memory_monitor.restart_requested(): + log.warning( + "Memory limit exceeded during sleep - initiating restart" + ) + break + + except asyncio.TimeoutError: + # Normal timeout, continue processing + pass + + continue + + log.info("Processing %s rows.", len(rows)) + empty_cycles = 0 + + # Reset delay and log if it was previously increased + if delay > self.interval_min: + log.info( + "Resetting delay from %s to %s seconds due to incoming events.", + delay, + self.interval_min, + ) + delay = self.interval_min + + # Process ALL claimed rows - but check for shutdown between batches + processed_ids = [] + for i, row in enumerate(rows): + # Check for shutdown signal periodically during processing + if i % 10 == 0 and self.shutdown_handler.shutdown_requested(): + log.info( + "Shutdown signal received during processing, finishing current batch..." + ) + + log.info( + "Processing row %s with event type %s.", row[0], row[2] + ) + + # Process event through all handlers + success = await self._process_event(row) + + # Only mark as processed if all handlers succeeded + if success: + processed_ids.append(row[0]) + else: + log.warning( + f"Skipping event {row[0]} due to handler errors" + ) + + if processed_ids: + await db_pool.mark_processed(processed_ids) + log.info("Processed %s rows.", len(processed_ids)) + + # Log pool statistics periodically + if len(rows) > 0: + pool_stats = await db_pool.get_pool_stats() + log.info("Pool statistics", extra=pool_stats) + + # After processing batch, check if shutdown or restart was requested + if self.shutdown_handler.shutdown_requested(): + log.info("Shutdown signal received, exiting gracefully...") + break + + except Exception as e: + log.exception(f"Error in outbox poller: {e}") + raise + finally: + # Cleanup: stop memory monitoring + log.info("Cleaning up resources...") + await memory_monitor.stop() + if not memory_task.done(): + memory_task.cancel() + try: + await memory_task + except asyncio.CancelledError: + pass + + # If we reach here, check exit condition + if memory_monitor.restart_requested(): + log.info("Exiting for restart due to memory limit") + sys.exit(RESTART_EXIT_CODE) + elif self.shutdown_handler.shutdown_requested(): + log.info( + f"Exiting gracefully due to {self.shutdown_handler.signal_received}" + ) + sys.exit(0) + + +async def handle_row(event_data: OutboxEvent): + log.info("Publishing event", extra=event_data.to_dict()) + + # Let's add this to the test_outbox_events table using Django ORM + + return True + + +class Command(BaseCommand): + help = "Runs the memory-safe async outbox poller with connection pooling, signal handling, and auto-restart" + + def add_arguments(self, parser): + parser.add_argument( + "--memory-limit", + type=int, + default=MEMORY_LIMIT_MB, + help="Memory limit in MB", + ) + parser.add_argument( + "--batch-size", + type=int, + default=BATCH_SIZE, + help="Batch size", + ) + parser.add_argument( + "--interval-min", + type=float, + default=INTERVAL_MIN, + help="Minimum interval in seconds", + ) + parser.add_argument( + "--interval-max", + type=float, + default=INTERVAL_MAX, + help="Maximum interval in seconds", + ) + parser.add_argument( + "--memory-check-interval", + type=int, + default=MEMORY_CHECK_INTERVAL, + help="Memory check interval in seconds", + ) + + def handle(self, *args, **options): + batch_size = options["batch_size"] + interval_min = options["interval_min"] + interval_max = options["interval_max"] + memory_limit_mb = options["memory_limit"] + memory_check_interval = options["memory_check_interval"] + + try: + self.stdout.write( + self.style.SUCCESS( + f"Starting outbox poller with connection pooling and signal handling:\n" + f" - Batch size: {batch_size}\n" + f" - Interval min: {interval_min}s\n" + f" - Interval max: {interval_max}s\n" + f" - Memory limit: {memory_limit_mb}MB\n" + f" - Memory check interval: {memory_check_interval}s\n" + f" - Pool size: {POOL_SIZE}\n" + f" - Pool min size: {POOL_MIN_SIZE}\n" + f" - Pool max size: {POOL_MAX_SIZE}\n" + f" - Pool timeout: {POOL_TIMEOUT}s\n" + f" - Pool max idle: {POOL_MAX_IDLE}s\n" + f" - Pool max lifetime: {POOL_MAX_LIFETIME}s\n" + f" - Graceful shutdown timeout: {GRACEFUL_SHUTDOWN_TIMEOUT}s" + ) + ) + + # Initialize the poller with the given configuration + poller = OutboxPoller( + batch_size=batch_size, + interval_min=interval_min, + interval_max=interval_max, + memory_limit_mb=memory_limit_mb, + memory_check_interval=memory_check_interval, + ) + + # Register the handler function to be invoked with each event + poller.add_handler(handle_row) + + # Start the poller loop + asyncio.run(poller.start()) + + except SystemExit as e: + if e.code == RESTART_EXIT_CODE: + self.stdout.write(self.style.WARNING("Restarting outbox poller...")) + os.execv(sys.executable, [sys.executable] + sys.argv) + else: + self.stdout.write(self.style.SUCCESS("Outbox poller shutdown complete")) + sys.exit(e.code) + except KeyboardInterrupt: + # This should be rare now since we handle SIGINT properly + self.stdout.write( + self.style.WARNING("Keyboard interrupt received, shutting down...") + ) + sys.exit(0) + except Exception as e: + self.stdout.write(self.style.ERROR(f"Unexpected error: {e}")) + log.exception(f"Unexpected error in outbox poller: {e}") + sys.exit(1) diff --git a/apps/api/plane/event_stream/migrations/0001_initial.py b/apps/api/plane/event_stream/migrations/0001_initial.py new file mode 100644 index 0000000000..e59f769834 --- /dev/null +++ b/apps/api/plane/event_stream/migrations/0001_initial.py @@ -0,0 +1,217 @@ +# Generated by Django 4.2.22 on 2025-08-08 06:42 + +from django.db import migrations, models +import django.db.models.manager +import django.utils.timezone +import pgtrigger.compiler +import pgtrigger.migrations +import uuid + + +class Migration(migrations.Migration): + + initial = True + + dependencies = [ + ('db', '0100_profile_has_marketing_email_consent_and_more'), + ] + + operations = [ + migrations.CreateModel( + name='CycleIssueProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.cycleissue',), + ), + migrations.CreateModel( + name='IssueAssigneeProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.issueassignee',), + ), + migrations.CreateModel( + name='IssueAttachmentProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.fileasset',), + ), + migrations.CreateModel( + name='IssueCommentProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.issuecomment',), + ), + migrations.CreateModel( + name='IssueLabelProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.issuelabel',), + ), + migrations.CreateModel( + name='IssueLinkProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.issuelink',), + ), + migrations.CreateModel( + name='IssueProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.issue',), + managers=[ + ('issue_objects', django.db.models.manager.Manager()), + ], + ), + migrations.CreateModel( + name='IssueRelationProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.issuerelation',), + ), + migrations.CreateModel( + name='ModuleIssueProxy', + fields=[ + ], + options={ + 'proxy': True, + 'indexes': [], + 'constraints': [], + }, + bases=('db.moduleissue',), + ), + migrations.CreateModel( + name='Outbox', + fields=[ + ('id', models.BigAutoField(primary_key=True, serialize=False)), + ('event_id', models.UUIDField(default=uuid.uuid4, editable=False, unique=True)), + ('event_type', models.CharField(max_length=255)), + ('entity_type', models.CharField(max_length=255)), + ('entity_id', models.UUIDField()), + ('payload', models.JSONField()), + ('processed_at', models.DateTimeField(blank=True, null=True)), + ('created_at', models.DateTimeField(default=django.utils.timezone.now)), + ('claimed_at', models.DateTimeField(blank=True, null=True)), + ('workspace_id', models.UUIDField()), + ('project_id', models.UUIDField()), + ('initiator_id', models.UUIDField(help_text='The user ID who triggered the event')), + ('initiator_type', models.CharField(default='USER', max_length=255)), + ], + options={ + 'db_table': 'outbox', + 'ordering': ['-created_at'], + 'indexes': [models.Index(condition=models.Q(('claimed_at__isnull', True), ('processed_at__isnull', True)), fields=['claimed_at', 'processed_at', 'id'], name='outbox_unclaimed_unprocessed'), models.Index(condition=models.Q(('processed_at__isnull', False)), fields=['processed_at'], name='outbox_processed_idx')], + }, + ), + pgtrigger.migrations.AddTrigger( + model_name='cycleissueproxy', + trigger=pgtrigger.compiler.Trigger(name='cycle_issue_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n BEGIN\n BEGIN\n -- try to enqueue event; ignore dupes and handle unexpected errors\n INSERT INTO outbox (\n event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type\n )\n VALUES (\n gen_random_uuid(),\n 'issue.cycle.added',\n 'cycle_issue',\n NEW.issue_id,\n jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'),\n NEW.workspace_id,\n NEW.project_id,\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING; -- (optional) skip duplicates\n EXCEPTION\n WHEN others THEN\n -- log but DO NOT re-throw, so the main insert survives\n RAISE WARNING 'Outbox insert failed for cycle_issue %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='565f83867cb222e250f01b77aa08146fedba230a', operation='INSERT', pgid='pgtrigger_cycle_issue_outbox_insert_66548', table='cycle_issues', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='cycleissueproxy', + trigger=pgtrigger.compiler.Trigger(name='cycle_issue_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n BEGIN\n BEGIN\n -- Check if this is a soft delete (deleted_at changed from null to not null)\n IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN\n -- This is a soft delete\n INSERT INTO outbox (\n event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type\n )\n VALUES (\n gen_random_uuid(),\n 'issue.cycle.removed',\n 'cycle_issue',\n OLD.issue_id,\n jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)),\n OLD.workspace_id,\n OLD.project_id,\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n ELSE\n -- This is a regular update - only trigger if cycle_id changed\n IF OLD.cycle_id IS DISTINCT FROM NEW.cycle_id THEN\n INSERT INTO outbox (\n event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type\n )\n VALUES (\n gen_random_uuid(),\n 'issue.cycle.moved',\n 'cycle_issue',\n NEW.issue_id,\n jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', row_to_json(OLD)),\n NEW.workspace_id,\n NEW.project_id,\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n END IF;\n END IF;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox update-event failed for cycle_issue %, reason: %',\n COALESCE(NEW.issue_id, OLD.issue_id), SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='b6ff117b5ac6b5347ea76fb9ea58d4b2f9d027b1', operation='UPDATE', pgid='pgtrigger_cycle_issue_outbox_update_38237', table='cycle_issues', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issueassigneeproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_assignee_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n enriched_data JSONB;\n assignee_ids JSONB;\n previous_assignee_ids JSONB;\n label_ids JSONB;\n issue_data JSONB;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.assignee.added'\n WHEN it.is_epic = true THEN 'epic.assignee.added'\n ELSE 'issue.assignee.added'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.assignee.added'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.assignee.added';\n END IF;\n \n -- Get the complete issue data (excluding description fields)\n SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'\n INTO issue_data\n FROM issues i\n WHERE i.id = NEW.issue_id;\n \n -- Fetch ALL current assignee IDs (including the newly added one)\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL;\n \n -- Fetch previous assignee IDs (excluding the newly added one)\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO previous_assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL AND ia.id != NEW.id;\n \n -- Fetch label IDs\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL;\n \n -- Create enriched data with complete issue, ALL assignee IDs (including new one), and label IDs\n enriched_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids\n );\n \n -- Create previous attributes with assignee IDs that existed before the addition\n issue_data := issue_data || jsonb_build_object(\n 'assignee_ids', previous_assignee_ids,\n 'label_ids', label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_assignee',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object('data', enriched_data, 'previous_attributes', issue_data),\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for issue_assignee %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='34aab09d0dad0a9eb732840002f423117fe42cbb', operation='INSERT', pgid='pgtrigger_issue_assignee_outbox_insert_51e4c', table='issue_assignees', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issueassigneeproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_assignee_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n enriched_data JSONB;\n current_assignee_ids JSONB;\n previous_assignee_ids JSONB;\n label_ids JSONB;\n issue_data JSONB;\n previous_issue_data JSONB;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.assignee.removed'\n WHEN it.is_epic = true THEN 'epic.assignee.removed'\n ELSE 'issue.assignee.removed'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = OLD.issue_id;\n \n -- If no issue found, default to 'issue.assignee.removed'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.assignee.removed';\n END IF;\n \n -- Get the complete issue data (excluding description fields)\n SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'\n INTO issue_data\n FROM issues i\n WHERE i.id = OLD.issue_id;\n \n -- Fetch current assignee IDs (excluding the one being removed)\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO current_assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL AND ia.id != OLD.id;\n \n -- Fetch ALL assignee IDs (including the one being removed) for previous_attributes\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO previous_assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL;\n \n -- Fetch label IDs\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL;\n \n -- Create enriched data with complete issue and current assignee IDs (after removal)\n enriched_data := issue_data || jsonb_build_object(\n 'assignee_ids', current_assignee_ids,\n 'label_ids', label_ids\n );\n \n -- Create previous attributes with complete issue and ALL assignee IDs (including the removed one)\n previous_issue_data := issue_data || jsonb_build_object(\n 'assignee_ids', previous_assignee_ids,\n 'label_ids', label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_assignee',\n OLD.issue_id,\n OLD.workspace_id,\n OLD.project_id,\n jsonb_build_object('data', enriched_data, 'previous_attributes', previous_issue_data),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox delete-event failed for issue_assignee %, reason: %',\n OLD.issue_id, SQLERRM;\n END;\n RETURN OLD;\n END;\n ", hash='eaf77930501ace8c7974991b9a86be648dee272d', operation='UPDATE', pgid='pgtrigger_issue_assignee_outbox_update_97b55', table='issue_assignees', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issueattachmentproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_attachment_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n BEGIN\n -- Only trigger for ISSUE_ATTACHMENT entity type\n IF NEW.entity_type = 'ISSUE_ATTACHMENT' THEN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.attachment.added'\n WHEN it.is_epic = true THEN 'epic.attachment.added'\n ELSE 'issue.attachment.added'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.attachment.added'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.attachment.added';\n END IF;\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_attachment',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'),\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for issue_attachment %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n END IF;\n RETURN NEW;\n END;\n ", hash='dfd7cc98782f6f0347bf92d36a021e45899d8c29', operation='INSERT', pgid='pgtrigger_issue_attachment_outbox_insert_b1598', table='file_assets', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issueattachmentproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_attachment_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n changes JSONB := '{}';\n field_name TEXT;\n old_value TEXT;\n new_value TEXT;\n event_type_name TEXT;\n BEGIN\n -- Only trigger for ISSUE_ATTACHMENT entity type\n IF NEW.entity_type = 'ISSUE_ATTACHMENT' OR OLD.entity_type = 'ISSUE_ATTACHMENT' THEN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.attachment.updated'\n WHEN it.is_epic = true THEN 'epic.attachment.updated'\n ELSE 'issue.attachment.updated'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.attachment.updated'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.attachment.updated';\n END IF;\n \n -- Check if this is a soft delete (deleted_at changed from null to not null)\n IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN\n -- This is a soft delete\n -- Determine delete event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.attachment.removed'\n WHEN it.is_epic = true THEN 'epic.attachment.removed'\n ELSE 'issue.attachment.removed'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = OLD.issue_id;\n \n -- If no issue found, default to 'issue.attachment.removed'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.attachment.removed';\n END IF;\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_attachment',\n OLD.issue_id,\n OLD.workspace_id,\n OLD.project_id,\n jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox delete failed for issue_attachment %, reason: %',\n OLD.issue_id, SQLERRM;\n END;\n ELSE\n -- This is a regular update, check for changes\n -- Loop through all columns to detect changes\n FOR field_name IN \n SELECT column_name \n FROM information_schema.columns \n WHERE table_name = 'file_assets' \n AND table_schema = 'public'\n AND column_name != 'updated_at' -- Skip updated_at column\n LOOP\n -- Get old and new values as text to avoid JSON conversion issues\n EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) \n INTO old_value, new_value \n USING OLD, NEW;\n \n -- If values are different, add to changes\n IF old_value IS DISTINCT FROM new_value THEN\n changes := changes || jsonb_build_object(\n field_name, \n old_value\n );\n END IF;\n END LOOP;\n \n -- Only insert if there are actual changes\n IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN\n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name, \n 'issue_attachment',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object(\n 'data', row_to_json(NEW),\n 'previous_attributes', changes\n ),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox update failed for issue_attachment %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n END IF;\n END IF;\n END IF;\n \n RETURN NEW;\n END;\n ", hash='b7cdd5f4da351e7d3b97a99b1212efcb1093a85b', operation='UPDATE', pgid='pgtrigger_issue_attachment_outbox_update_5fc41', table='file_assets', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuecommentproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_comment_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n enriched_data JSONB;\n assignee_ids JSONB;\n label_ids JSONB;\n issue_data JSONB;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.comment.created'\n WHEN it.is_epic = true THEN 'epic.comment.created'\n ELSE 'issue.comment.created'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.comment.created'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.comment.created';\n END IF;\n \n -- Get the complete issue data (excluding description fields)\n SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'\n INTO issue_data\n FROM issues i\n WHERE i.id = NEW.issue_id;\n \n -- Fetch assignee IDs\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL;\n \n -- Fetch label IDs\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL;\n \n -- Create enriched data with complete issue, assignee IDs, and label IDs\n enriched_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids,\n 'comment', row_to_json(NEW)\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object('data', enriched_data, 'previous_attributes', '{}'),\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for issue_comment %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='2e57bd2fe5275c8fef7bb8a1f318bea7bfc78c76', operation='INSERT', pgid='pgtrigger_issue_comment_outbox_insert_bdd46', table='issue_comments', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuecommentproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_comment_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n changes JSONB := '{}';\n field_name TEXT;\n old_value TEXT;\n new_value TEXT;\n event_type_name TEXT;\n enriched_data JSONB;\n assignee_ids JSONB;\n label_ids JSONB;\n issue_data JSONB;\n BEGIN\n -- Check if this is a soft delete (deleted_at changed from null to not null)\n IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN\n -- This is a soft delete\n -- Determine delete event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.comment.deleted'\n WHEN it.is_epic = true THEN 'epic.comment.deleted'\n ELSE 'issue.comment.deleted'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = OLD.issue_id;\n \n -- If no issue found, default to 'issue.comment.deleted'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.comment.deleted';\n END IF;\n \n -- Get the complete issue data (excluding description fields)\n SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'\n INTO issue_data\n FROM issues i\n WHERE i.id = OLD.issue_id;\n \n -- Fetch assignee IDs\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL;\n \n -- Fetch label IDs\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL;\n \n -- Create enriched data with complete issue, assignee IDs, and label IDs\n enriched_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue',\n OLD.issue_id,\n OLD.workspace_id,\n OLD.project_id,\n jsonb_build_object('data', enriched_data, 'previous_attributes', row_to_json(OLD)),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox delete failed for issue_comment %, reason: %',\n OLD.issue_id, SQLERRM;\n END;\n ELSE\n -- This is a regular update, check for changes\n -- Determine update event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.comment.updated'\n WHEN it.is_epic = true THEN 'epic.comment.updated'\n ELSE 'issue.comment.updated'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.comment.updated'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.comment.updated';\n END IF;\n \n -- Loop through all columns to detect changes\n FOR field_name IN \n SELECT column_name \n FROM information_schema.columns \n WHERE table_name = 'issue_comments' \n AND table_schema = 'public'\n AND column_name != 'updated_at' -- Skip updated_at column\n LOOP\n -- Get old and new values as text to avoid JSON conversion issues\n EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) \n INTO old_value, new_value \n USING OLD, NEW;\n \n -- If values are different, add to changes\n IF old_value IS DISTINCT FROM new_value THEN\n changes := changes || jsonb_build_object(\n field_name, \n old_value\n );\n END IF;\n END LOOP;\n \n -- Only insert if there are actual changes\n IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN\n -- Get the complete issue data (excluding description fields)\n SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'\n INTO issue_data\n FROM issues i\n WHERE i.id = NEW.issue_id;\n \n -- Fetch assignee IDs\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL;\n \n -- Fetch label IDs\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL;\n \n -- Create enriched data with complete issue, assignee IDs, label IDs, and updated comment\n enriched_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids,\n 'comment', row_to_json(NEW)\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name, \n 'issue',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object(\n 'data', enriched_data,\n 'previous_attributes', changes\n ),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox update failed for issue_comment %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n END IF;\n END IF;\n \n RETURN NEW;\n END;\n ", hash='899438c837ab637cc928d12e382abaadd8ea36a1', operation='UPDATE', pgid='pgtrigger_issue_comment_outbox_update_dc7b2', table='issue_comments', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuelabelproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_label_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n enriched_data JSONB;\n assignee_ids JSONB;\n label_ids JSONB;\n previous_label_ids JSONB;\n issue_data JSONB;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.label.added'\n WHEN it.is_epic = true THEN 'epic.label.added'\n ELSE 'issue.label.added'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.label.added'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.label.added';\n END IF;\n \n -- Get the complete issue data (excluding description fields)\n SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'\n INTO issue_data\n FROM issues i\n WHERE i.id = NEW.issue_id;\n \n -- Fetch assignee IDs\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL;\n \n -- Fetch ALL current label IDs (including the newly added one)\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL;\n \n -- Fetch previous label IDs (excluding the newly added one)\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO previous_label_ids\n FROM issue_labels il\n WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL AND il.id != NEW.id;\n \n -- Create enriched data with complete issue, assignee IDs, and ALL label IDs (including new one)\n enriched_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids\n );\n \n -- Create previous attributes with label IDs that existed before the addition\n issue_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', previous_label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_label',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object('data', enriched_data, 'previous_attributes', issue_data),\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for issue_label %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='e145311ba4d88aa53755fb392e26d2539b67faf4', operation='INSERT', pgid='pgtrigger_issue_label_outbox_insert_a8b0d', table='issue_labels', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuelabelproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_label_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n enriched_data JSONB;\n assignee_ids JSONB;\n current_label_ids JSONB;\n previous_label_ids JSONB;\n issue_data JSONB;\n previous_issue_data JSONB;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.label.removed'\n WHEN it.is_epic = true THEN 'epic.label.removed'\n ELSE 'issue.label.removed'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = OLD.issue_id;\n \n -- If no issue found, default to 'issue.label.removed'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.label.removed';\n END IF;\n \n -- Get the complete issue data (excluding description fields)\n SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'\n INTO issue_data\n FROM issues i\n WHERE i.id = OLD.issue_id;\n \n -- Fetch assignee IDs\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL;\n \n -- Fetch current label IDs (excluding the one being removed)\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO current_label_ids\n FROM issue_labels il\n WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL AND il.id != OLD.id;\n \n -- Fetch ALL label IDs (including the one being removed) for previous_attributes\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO previous_label_ids\n FROM issue_labels il\n WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL;\n \n -- Create enriched data with complete issue, assignee IDs, and current label IDs (after removal)\n enriched_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', current_label_ids\n );\n \n -- Create previous attributes with complete issue, assignee IDs, and ALL label IDs (including the removed one)\n previous_issue_data := issue_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', previous_label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_label',\n OLD.issue_id,\n OLD.workspace_id,\n OLD.project_id,\n jsonb_build_object('data', enriched_data, 'previous_attributes', previous_issue_data),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox delete-event failed for issue_label %, reason: %',\n OLD.issue_id, SQLERRM;\n END;\n RETURN OLD;\n END;\n ", hash='da94f8d0b65d794bbdcd02ad9c32e405a31ed823', operation='UPDATE', pgid='pgtrigger_issue_label_outbox_update_47e56', table='issue_labels', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuelinkproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_link_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.link.added'\n WHEN it.is_epic = true THEN 'epic.link.added'\n ELSE 'issue.link.added'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.link.added'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.link.added';\n END IF;\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_link',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'),\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for issue_link %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='6a9beceb948afcc96af6431bd69e8c623a10af46', operation='INSERT', pgid='pgtrigger_issue_link_outbox_insert_38eb1', table='issue_links', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuelinkproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_link_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n changes JSONB := '{}';\n field_name TEXT;\n old_value TEXT;\n new_value TEXT;\n event_type_name TEXT;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.link.updated'\n WHEN it.is_epic = true THEN 'epic.link.updated'\n ELSE 'issue.link.updated'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.link.updated'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.link.updated';\n END IF;\n \n -- Check if this is a soft delete (deleted_at changed from null to not null)\n IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN\n -- This is a soft delete\n -- Determine delete event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.link.removed'\n WHEN it.is_epic = true THEN 'epic.link.removed'\n ELSE 'issue.link.removed'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = OLD.issue_id;\n \n -- If no issue found, default to 'issue.link.removed'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.link.removed';\n END IF;\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue',\n OLD.issue_id,\n OLD.workspace_id,\n OLD.project_id,\n jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox delete failed for issue %, reason: %',\n OLD.id, SQLERRM;\n END;\n ELSE\n -- This is a regular update, check for changes\n -- Loop through all columns to detect changes\n FOR field_name IN \n SELECT column_name \n FROM information_schema.columns \n WHERE table_name = 'issue_links' \n AND table_schema = 'public'\n AND column_name != 'updated_at' -- Skip updated_at column\n LOOP\n -- Get old and new values as text to avoid JSON conversion issues\n EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) \n INTO old_value, new_value \n USING OLD, NEW;\n \n -- If values are different, add to changes\n IF old_value IS DISTINCT FROM new_value THEN\n changes := changes || jsonb_build_object(\n field_name, \n old_value\n );\n END IF;\n END LOOP;\n \n -- Only insert if there are actual changes\n IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN\n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name, \n 'issue',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object(\n 'data', row_to_json(NEW),\n 'previous_attributes', changes\n ),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox update failed for issue %, reason: %',\n NEW.id, SQLERRM;\n END;\n END IF;\n END IF;\n \n RETURN NEW;\n END;\n ", hash='1676b5c913cf9212967a99da830af396e4a58fd4', operation='UPDATE', pgid='pgtrigger_issue_link_outbox_update_0b646', table='issue_links', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issueproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n filtered_new_data JSONB;\n assignee_ids JSONB;\n label_ids JSONB;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN NEW.type_id IS NULL THEN 'issue.created'\n WHEN it.is_epic = true THEN 'epic.created'\n ELSE 'issue.created'\n END\n INTO event_type_name\n FROM issue_types it\n WHERE it.id = NEW.type_id;\n \n -- If no issue type found, default to 'issue.created'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.created';\n END IF;\n \n -- Fetch assignee IDs\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = NEW.id AND ia.deleted_at IS NULL;\n \n -- Fetch label IDs\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = NEW.id AND il.deleted_at IS NULL;\n \n -- Create filtered NEW data excluding description fields\n filtered_new_data := to_jsonb(NEW) - 'description_html' - 'description_binary' - 'description' - 'description_stripped';\n \n -- Add assignee and label IDs to the data\n filtered_new_data := filtered_new_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue',\n NEW.id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object(\n 'data', filtered_new_data,\n 'previous_attributes', '{}'\n ),\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for issue %, reason: %',\n NEW.id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='346a4d578b066e3933a44ce1cfdaf6ac662195e3', operation='INSERT', pgid='pgtrigger_issue_outbox_insert_0de30', table='issues', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issueproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n changes JSONB := '{}';\n field_name TEXT;\n old_value TEXT;\n new_value TEXT;\n event_type_name TEXT;\n old_is_epic BOOLEAN := false;\n new_is_epic BOOLEAN := false;\n conversion_event_name TEXT;\n filtered_new_data JSONB;\n filtered_old_data JSONB;\n assignee_ids JSONB;\n label_ids JSONB;\n has_non_description_changes BOOLEAN := false;\n state_changed BOOLEAN := false;\n BEGIN\n -- Check if this is a soft delete (deleted_at changed from null to not null)\n IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN\n -- This is a soft delete\n -- Determine delete event type based on issue type\n SELECT \n CASE \n WHEN OLD.type_id IS NULL THEN 'issue.deleted'\n WHEN it.is_epic = true THEN 'epic.deleted'\n ELSE 'issue.deleted'\n END\n INTO event_type_name\n FROM issue_types it\n WHERE it.id = OLD.type_id;\n \n -- If no issue type found, default to 'issue.deleted'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.deleted';\n END IF;\n \n -- Create filtered OLD data excluding description fields\n filtered_old_data := to_jsonb(OLD) - 'description_html' - 'description_binary' - 'description' - 'description_stripped';\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue',\n OLD.id,\n OLD.workspace_id,\n OLD.project_id,\n jsonb_build_object('data', '{}', 'previous_attributes', filtered_old_data),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox delete failed for issue %, reason: %',\n OLD.id, SQLERRM;\n END;\n ELSE\n -- This is a regular update, check for changes excluding description fields\n -- First, check if there are any non-description field changes\n FOR field_name IN \n SELECT column_name \n FROM information_schema.columns \n WHERE table_name = 'issues' \n AND table_schema = 'public'\n AND column_name NOT IN ('updated_at', 'updated_by_id', 'description_html', 'description_binary', 'description', 'description_stripped') -- Skip description fields\n LOOP\n -- Get old and new values as text to avoid JSON conversion issues\n EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) \n INTO old_value, new_value \n USING OLD, NEW;\n \n -- If values are different, add to changes and mark that we have non-description changes\n IF old_value IS DISTINCT FROM new_value THEN\n has_non_description_changes := true;\n changes := changes || jsonb_build_object(\n field_name, \n old_value\n );\n \n -- Check if state field has changed\n IF field_name = 'state_id' THEN\n state_changed := true;\n END IF;\n END IF;\n END LOOP;\n \n -- Only proceed if there are non-description field changes\n IF has_non_description_changes THEN\n -- Determine event type based on issue type and whether state changed\n IF state_changed THEN\n -- State has changed, use state-specific event types\n SELECT \n CASE \n WHEN NEW.type_id IS NULL THEN 'issue.state.updated'\n WHEN it.is_epic = true THEN 'epic.state.updated'\n ELSE 'issue.state.updated'\n END\n INTO event_type_name\n FROM issue_types it\n WHERE it.id = NEW.type_id;\n \n -- If no issue type found, default to 'issue.state.updated'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.state.updated';\n END IF;\n ELSE\n -- Regular update, use generic event types\n SELECT \n CASE \n WHEN NEW.type_id IS NULL THEN 'issue.updated'\n WHEN it.is_epic = true THEN 'epic.updated'\n ELSE 'issue.updated'\n END\n INTO event_type_name\n FROM issue_types it\n WHERE it.id = NEW.type_id;\n \n -- If no issue type found, default to 'issue.updated'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.updated';\n END IF;\n END IF;\n \n -- Fetch assignee IDs\n SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb)\n INTO assignee_ids\n FROM issue_assignees ia\n WHERE ia.issue_id = NEW.id AND ia.deleted_at IS NULL;\n \n -- Fetch label IDs\n SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb)\n INTO label_ids\n FROM issue_labels il\n WHERE il.issue_id = NEW.id AND il.deleted_at IS NULL;\n \n -- Create filtered NEW data excluding description fields\n filtered_new_data := to_jsonb(NEW) - 'description_html' - 'description_binary' - 'description' - 'description_stripped';\n \n -- Add assignee and label IDs to the data\n filtered_new_data := filtered_new_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name, \n 'issue',\n NEW.id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object(\n 'data', filtered_new_data,\n 'previous_attributes', changes\n ),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox update failed for issue %, reason: %',\n NEW.id, SQLERRM;\n END;\n END IF;\n \n -- Check if type_id has changed to trigger conversion events (only if there are other changes)\n IF has_non_description_changes AND OLD.type_id IS DISTINCT FROM NEW.type_id THEN\n -- Determine if old type was epic\n IF OLD.type_id IS NOT NULL THEN\n SELECT COALESCE(it.is_epic, false) INTO old_is_epic\n FROM issue_types it\n WHERE it.id = OLD.type_id;\n END IF;\n \n -- Determine if new type is epic\n IF NEW.type_id IS NOT NULL THEN\n SELECT COALESCE(it.is_epic, false) INTO new_is_epic\n FROM issue_types it\n WHERE it.id = NEW.type_id;\n END IF;\n \n -- Determine conversion event name\n IF old_is_epic = false AND new_is_epic = true THEN\n conversion_event_name := 'issue.converted.to_epic';\n ELSIF old_is_epic = true AND new_is_epic = false THEN\n conversion_event_name := 'epic.converted.to_issue';\n END IF;\n \n -- Insert conversion event if applicable\n IF conversion_event_name IS NOT NULL THEN\n -- Create filtered NEW data excluding description fields for conversion event\n filtered_new_data := to_jsonb(NEW) - 'description_html' - 'description_binary' - 'description' - 'description_stripped';\n \n -- Add assignee and label IDs to the conversion data\n filtered_new_data := filtered_new_data || jsonb_build_object(\n 'assignee_ids', assignee_ids,\n 'label_ids', label_ids\n );\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n conversion_event_name,\n 'issue',\n NEW.id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object(\n 'data', filtered_new_data,\n 'previous_attributes', jsonb_build_object('type_id', OLD.type_id)\n ),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox conversion failed for issue %, reason: %',\n NEW.id, SQLERRM;\n END;\n END IF;\n END IF;\n END IF;\n \n RETURN NEW;\n END;\n ", hash='cc98de56ebb2458a3a0ed344c2752e2c9c9e66a2', operation='UPDATE', pgid='pgtrigger_issue_outbox_update_b23e9', table='issues', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuerelationproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_relation_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n event_type_name TEXT;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.relation.added'\n WHEN it.is_epic = true THEN 'epic.relation.added'\n ELSE 'issue.relation.added'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.relation.added'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.relation.added';\n END IF;\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue_relation',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'),\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for issue_relation %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='9b750c88ffe1a1218dd0fc01e6fe719ddea41f7d', operation='INSERT', pgid='pgtrigger_issue_relation_outbox_insert_29038', table='issue_relations', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='issuerelationproxy', + trigger=pgtrigger.compiler.Trigger(name='issue_relation_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n DECLARE\n changes JSONB := '{}';\n field_name TEXT;\n old_value TEXT;\n new_value TEXT;\n event_type_name TEXT;\n BEGIN\n -- Determine event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.relation.updated'\n WHEN it.is_epic = true THEN 'epic.relation.updated'\n ELSE 'issue.relation.updated'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = NEW.issue_id;\n \n -- If no issue found, default to 'issue.relation.updated'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.relation.updated';\n END IF;\n \n -- Check if this is a soft delete (deleted_at changed from null to not null)\n IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN\n -- This is a soft delete\n -- Determine delete event type based on issue type\n SELECT \n CASE \n WHEN i.type_id IS NULL THEN 'issue.relation.removed'\n WHEN it.is_epic = true THEN 'epic.relation.removed'\n ELSE 'issue.relation.removed'\n END\n INTO event_type_name\n FROM issues i\n LEFT JOIN issue_types it ON it.id = i.type_id\n WHERE i.id = OLD.issue_id;\n \n -- If no issue found, default to 'issue.relation.removed'\n IF event_type_name IS NULL THEN\n event_type_name := 'issue.relation.removed';\n END IF;\n \n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name,\n 'issue',\n OLD.issue_id,\n OLD.workspace_id,\n OLD.project_id,\n jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox delete failed for issue_relation %, reason: %',\n OLD.issue_id, SQLERRM;\n END;\n ELSE\n -- This is a regular update, check for changes\n -- Loop through all columns to detect changes\n FOR field_name IN \n SELECT column_name \n FROM information_schema.columns \n WHERE table_name = 'issue_relations' \n AND table_schema = 'public'\n AND column_name != 'updated_at' -- Skip updated_at column\n LOOP\n -- Get old and new values as text to avoid JSON conversion issues\n EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) \n INTO old_value, new_value \n USING OLD, NEW;\n \n -- If values are different, add to changes\n IF old_value IS DISTINCT FROM new_value THEN\n changes := changes || jsonb_build_object(\n field_name, \n old_value\n );\n END IF;\n END LOOP;\n \n -- Only insert if there are actual changes\n IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN\n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n event_type_name, \n 'issue',\n NEW.issue_id,\n NEW.workspace_id,\n NEW.project_id,\n jsonb_build_object(\n 'data', row_to_json(NEW),\n 'previous_attributes', changes\n ),\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox update failed for issue_relation %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n END IF;\n END IF;\n RETURN NEW;\n END;\n ", hash='9608c3f2dbed5321b912126156537518d34d9c6a', operation='UPDATE', pgid='pgtrigger_issue_relation_outbox_update_0c350', table='issue_relations', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='moduleissueproxy', + trigger=pgtrigger.compiler.Trigger(name='module_issue_outbox_insert', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n BEGIN\n BEGIN\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n 'issue.module.added',\n 'module_issue',\n NEW.issue_id,\n jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'),\n NEW.workspace_id,\n NEW.project_id,\n now(),\n NEW.created_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox insert failed for module_issue %, reason: %',\n NEW.issue_id, SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='765a7bd190447461b57e28cbc2a5047d2973e059', operation='INSERT', pgid='pgtrigger_module_issue_outbox_insert_4a4e3', table='module_issues', when='AFTER')), + ), + pgtrigger.migrations.AddTrigger( + model_name='moduleissueproxy', + trigger=pgtrigger.compiler.Trigger(name='module_issue_outbox_update', sql=pgtrigger.compiler.UpsertTriggerSql(func="\n BEGIN\n BEGIN\n -- Check if this is a soft delete (deleted_at changed from null to not null)\n IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN\n -- This is a soft delete\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n 'issue.module.removed',\n 'module_issue',\n OLD.issue_id,\n jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)),\n OLD.workspace_id,\n OLD.project_id,\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n ELSE\n -- This is a regular update\n INSERT INTO outbox (event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type)\n VALUES (\n gen_random_uuid(),\n 'issue.module.moved',\n 'module_issue',\n NEW.issue_id,\n jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', row_to_json(OLD)),\n NEW.workspace_id,\n NEW.project_id,\n now(),\n NEW.updated_by_id,\n COALESCE(current_setting('plane.initiator_type', true), 'USER')\n )\n ON CONFLICT DO NOTHING;\n END IF;\n EXCEPTION\n WHEN others THEN\n RAISE WARNING 'Outbox update-event failed for module_issue %, reason: %',\n COALESCE(NEW.issue_id, OLD.issue_id), SQLERRM;\n END;\n RETURN NEW;\n END;\n ", hash='7cc218096c880a14df92c89bdfa84b82e619049e', operation='UPDATE', pgid='pgtrigger_module_issue_outbox_update_9f191', table='module_issues', when='AFTER')), + ), + ] diff --git a/apps/api/plane/event_stream/migrations/__init__.py b/apps/api/plane/event_stream/migrations/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/apps/api/plane/event_stream/models/__init__.py b/apps/api/plane/event_stream/models/__init__.py new file mode 100644 index 0000000000..311c733bbf --- /dev/null +++ b/apps/api/plane/event_stream/models/__init__.py @@ -0,0 +1,4 @@ +from .outbox import Outbox +from .issue import IssueProxy +from .cycle import CycleIssueProxy +from .module import ModuleIssueProxy diff --git a/apps/api/plane/event_stream/models/cycle.py b/apps/api/plane/event_stream/models/cycle.py new file mode 100644 index 0000000000..a995fb3a12 --- /dev/null +++ b/apps/api/plane/event_stream/models/cycle.py @@ -0,0 +1,102 @@ +import pgtrigger +from plane.db.models import CycleIssue + + +class CycleIssueProxy(CycleIssue): + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="cycle_issue_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + BEGIN + BEGIN + -- try to enqueue event; ignore dupes and handle unexpected errors + INSERT INTO outbox ( + event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type + ) + VALUES ( + gen_random_uuid(), + 'issue.cycle.added', + 'cycle_issue', + NEW.issue_id, + jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'), + NEW.workspace_id, + NEW.project_id, + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; -- (optional) skip duplicates + EXCEPTION + WHEN others THEN + -- log but DO NOT re-throw, so the main insert survives + RAISE WARNING 'Outbox insert failed for cycle_issue %, reason: %', + NEW.issue_id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + # Handle both soft deletes (deleted_at updated to not null) and regular updates + pgtrigger.Trigger( + name="cycle_issue_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + BEGIN + BEGIN + -- Check if this is a soft delete (deleted_at changed from null to not null) + IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN + -- This is a soft delete + INSERT INTO outbox ( + event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type + ) + VALUES ( + gen_random_uuid(), + 'issue.cycle.removed', + 'cycle_issue', + OLD.issue_id, + jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)), + OLD.workspace_id, + OLD.project_id, + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + ELSE + -- This is a regular update - only trigger if cycle_id changed + IF OLD.cycle_id IS DISTINCT FROM NEW.cycle_id THEN + INSERT INTO outbox ( + event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type + ) + VALUES ( + gen_random_uuid(), + 'issue.cycle.moved', + 'cycle_issue', + NEW.issue_id, + jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', row_to_json(OLD)), + NEW.workspace_id, + NEW.project_id, + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + END IF; + END IF; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox update-event failed for cycle_issue %, reason: %', + COALESCE(NEW.issue_id, OLD.issue_id), SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + ] diff --git a/apps/api/plane/event_stream/models/issue.py b/apps/api/plane/event_stream/models/issue.py new file mode 100644 index 0000000000..5163d046a8 --- /dev/null +++ b/apps/api/plane/event_stream/models/issue.py @@ -0,0 +1,1538 @@ +import pgtrigger +from plane.db.models import ( + Issue, + IssueAssignee, + IssueLabel, + IssueComment, + IssueLink, + IssueRelation, + FileAsset, +) + + +class IssueProxy(Issue): + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="issue_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + filtered_new_data JSONB; + assignee_ids JSONB; + label_ids JSONB; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN NEW.type_id IS NULL THEN 'issue.created' + WHEN it.is_epic = true THEN 'epic.created' + ELSE 'issue.created' + END + INTO event_type_name + FROM issue_types it + WHERE it.id = NEW.type_id; + + -- If no issue type found, default to 'issue.created' + IF event_type_name IS NULL THEN + event_type_name := 'issue.created'; + END IF; + + -- Fetch assignee IDs + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = NEW.id AND ia.deleted_at IS NULL; + + -- Fetch label IDs + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = NEW.id AND il.deleted_at IS NULL; + + -- Create filtered NEW data excluding description fields + filtered_new_data := to_jsonb(NEW) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'; + + -- Add assignee and label IDs to the data + filtered_new_data := filtered_new_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + NEW.id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object( + 'data', filtered_new_data, + 'previous_attributes', '{}' + ), + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for issue %, reason: %', + NEW.id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + pgtrigger.Trigger( + name="issue_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + DECLARE + changes JSONB := '{}'; + field_name TEXT; + old_value TEXT; + new_value TEXT; + event_type_name TEXT; + old_is_epic BOOLEAN := false; + new_is_epic BOOLEAN := false; + conversion_event_name TEXT; + filtered_new_data JSONB; + filtered_old_data JSONB; + assignee_ids JSONB; + label_ids JSONB; + has_non_description_changes BOOLEAN := false; + state_changed BOOLEAN := false; + BEGIN + -- Check if this is a soft delete (deleted_at changed from null to not null) + IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN + -- This is a soft delete + -- Determine delete event type based on issue type + SELECT + CASE + WHEN OLD.type_id IS NULL THEN 'issue.deleted' + WHEN it.is_epic = true THEN 'epic.deleted' + ELSE 'issue.deleted' + END + INTO event_type_name + FROM issue_types it + WHERE it.id = OLD.type_id; + + -- If no issue type found, default to 'issue.deleted' + IF event_type_name IS NULL THEN + event_type_name := 'issue.deleted'; + END IF; + + -- Create filtered OLD data excluding description fields + filtered_old_data := to_jsonb(OLD) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'; + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + OLD.id, + OLD.workspace_id, + OLD.project_id, + jsonb_build_object('data', '{}', 'previous_attributes', filtered_old_data), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox delete failed for issue %, reason: %', + OLD.id, SQLERRM; + END; + ELSE + -- This is a regular update, check for changes excluding description fields + -- First, check if there are any non-description field changes + FOR field_name IN + SELECT column_name + FROM information_schema.columns + WHERE table_name = 'issues' + AND table_schema = 'public' + AND column_name NOT IN ('updated_at', 'updated_by_id', 'description_html', 'description_binary', 'description', 'description_stripped') -- Skip description fields + LOOP + -- Get old and new values as text to avoid JSON conversion issues + EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) + INTO old_value, new_value + USING OLD, NEW; + + -- If values are different, add to changes and mark that we have non-description changes + IF old_value IS DISTINCT FROM new_value THEN + has_non_description_changes := true; + changes := changes || jsonb_build_object( + field_name, + old_value + ); + + -- Check if state field has changed + IF field_name = 'state_id' THEN + state_changed := true; + END IF; + END IF; + END LOOP; + + -- Only proceed if there are non-description field changes + IF has_non_description_changes THEN + -- Determine event type based on issue type and whether state changed + IF state_changed THEN + -- State has changed, use state-specific event types + SELECT + CASE + WHEN NEW.type_id IS NULL THEN 'issue.state.updated' + WHEN it.is_epic = true THEN 'epic.state.updated' + ELSE 'issue.state.updated' + END + INTO event_type_name + FROM issue_types it + WHERE it.id = NEW.type_id; + + -- If no issue type found, default to 'issue.state.updated' + IF event_type_name IS NULL THEN + event_type_name := 'issue.state.updated'; + END IF; + ELSE + -- Regular update, use generic event types + SELECT + CASE + WHEN NEW.type_id IS NULL THEN 'issue.updated' + WHEN it.is_epic = true THEN 'epic.updated' + ELSE 'issue.updated' + END + INTO event_type_name + FROM issue_types it + WHERE it.id = NEW.type_id; + + -- If no issue type found, default to 'issue.updated' + IF event_type_name IS NULL THEN + event_type_name := 'issue.updated'; + END IF; + END IF; + + -- Fetch assignee IDs + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = NEW.id AND ia.deleted_at IS NULL; + + -- Fetch label IDs + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = NEW.id AND il.deleted_at IS NULL; + + -- Create filtered NEW data excluding description fields + filtered_new_data := to_jsonb(NEW) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'; + + -- Add assignee and label IDs to the data + filtered_new_data := filtered_new_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + NEW.id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object( + 'data', filtered_new_data, + 'previous_attributes', changes + ), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox update failed for issue %, reason: %', + NEW.id, SQLERRM; + END; + END IF; + + -- Check if type_id has changed to trigger conversion events (only if there are other changes) + IF has_non_description_changes AND OLD.type_id IS DISTINCT FROM NEW.type_id THEN + -- Determine if old type was epic + IF OLD.type_id IS NOT NULL THEN + SELECT COALESCE(it.is_epic, false) INTO old_is_epic + FROM issue_types it + WHERE it.id = OLD.type_id; + END IF; + + -- Determine if new type is epic + IF NEW.type_id IS NOT NULL THEN + SELECT COALESCE(it.is_epic, false) INTO new_is_epic + FROM issue_types it + WHERE it.id = NEW.type_id; + END IF; + + -- Determine conversion event name + IF old_is_epic = false AND new_is_epic = true THEN + conversion_event_name := 'issue.converted.to_epic'; + ELSIF old_is_epic = true AND new_is_epic = false THEN + conversion_event_name := 'epic.converted.to_issue'; + END IF; + + -- Insert conversion event if applicable + IF conversion_event_name IS NOT NULL THEN + -- Create filtered NEW data excluding description fields for conversion event + filtered_new_data := to_jsonb(NEW) - 'description_html' - 'description_binary' - 'description' - 'description_stripped'; + + -- Add assignee and label IDs to the conversion data + filtered_new_data := filtered_new_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + conversion_event_name, + 'issue', + NEW.id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object( + 'data', filtered_new_data, + 'previous_attributes', jsonb_build_object('type_id', OLD.type_id) + ), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox conversion failed for issue %, reason: %', + NEW.id, SQLERRM; + END; + END IF; + END IF; + END IF; + + RETURN NEW; + END; + """, + condition=None, + ), + ] + + +class IssueAssigneeProxy(IssueAssignee): + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="issue_assignee_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + enriched_data JSONB; + assignee_ids JSONB; + previous_assignee_ids JSONB; + label_ids JSONB; + issue_data JSONB; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.assignee.added' + WHEN it.is_epic = true THEN 'epic.assignee.added' + ELSE 'issue.assignee.added' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.assignee.added' + IF event_type_name IS NULL THEN + event_type_name := 'issue.assignee.added'; + END IF; + + -- Get the complete issue data (excluding description fields) + SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped' + INTO issue_data + FROM issues i + WHERE i.id = NEW.issue_id; + + -- Fetch ALL current assignee IDs (including the newly added one) + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL; + + -- Fetch previous assignee IDs (excluding the newly added one) + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO previous_assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL AND ia.id != NEW.id; + + -- Fetch label IDs + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL; + + -- Create enriched data with complete issue, ALL assignee IDs (including new one), and label IDs + enriched_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids + ); + + -- Create previous attributes with assignee IDs that existed before the addition + issue_data := issue_data || jsonb_build_object( + 'assignee_ids', previous_assignee_ids, + 'label_ids', label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_assignee', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object('data', enriched_data, 'previous_attributes', issue_data), + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for issue_assignee %, reason: %', + NEW.issue_id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + # Since we do a soft delete, we need to update the outbox when the assignee is removed + pgtrigger.Trigger( + name="issue_assignee_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + enriched_data JSONB; + current_assignee_ids JSONB; + previous_assignee_ids JSONB; + label_ids JSONB; + issue_data JSONB; + previous_issue_data JSONB; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.assignee.removed' + WHEN it.is_epic = true THEN 'epic.assignee.removed' + ELSE 'issue.assignee.removed' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = OLD.issue_id; + + -- If no issue found, default to 'issue.assignee.removed' + IF event_type_name IS NULL THEN + event_type_name := 'issue.assignee.removed'; + END IF; + + -- Get the complete issue data (excluding description fields) + SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped' + INTO issue_data + FROM issues i + WHERE i.id = OLD.issue_id; + + -- Fetch current assignee IDs (excluding the one being removed) + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO current_assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL AND ia.id != OLD.id; + + -- Fetch ALL assignee IDs (including the one being removed) for previous_attributes + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO previous_assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL; + + -- Fetch label IDs + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL; + + -- Create enriched data with complete issue and current assignee IDs (after removal) + enriched_data := issue_data || jsonb_build_object( + 'assignee_ids', current_assignee_ids, + 'label_ids', label_ids + ); + + -- Create previous attributes with complete issue and ALL assignee IDs (including the removed one) + previous_issue_data := issue_data || jsonb_build_object( + 'assignee_ids', previous_assignee_ids, + 'label_ids', label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_assignee', + OLD.issue_id, + OLD.workspace_id, + OLD.project_id, + jsonb_build_object('data', enriched_data, 'previous_attributes', previous_issue_data), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox delete-event failed for issue_assignee %, reason: %', + OLD.issue_id, SQLERRM; + END; + RETURN OLD; + END; + """, + condition=None, + ), + ] + + +class IssueLabelProxy(IssueLabel): + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="issue_label_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + enriched_data JSONB; + assignee_ids JSONB; + label_ids JSONB; + previous_label_ids JSONB; + issue_data JSONB; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.label.added' + WHEN it.is_epic = true THEN 'epic.label.added' + ELSE 'issue.label.added' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.label.added' + IF event_type_name IS NULL THEN + event_type_name := 'issue.label.added'; + END IF; + + -- Get the complete issue data (excluding description fields) + SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped' + INTO issue_data + FROM issues i + WHERE i.id = NEW.issue_id; + + -- Fetch assignee IDs + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL; + + -- Fetch ALL current label IDs (including the newly added one) + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL; + + -- Fetch previous label IDs (excluding the newly added one) + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO previous_label_ids + FROM issue_labels il + WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL AND il.id != NEW.id; + + -- Create enriched data with complete issue, assignee IDs, and ALL label IDs (including new one) + enriched_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids + ); + + -- Create previous attributes with label IDs that existed before the addition + issue_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', previous_label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_label', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object('data', enriched_data, 'previous_attributes', issue_data), + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for issue_label %, reason: %', + NEW.issue_id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + # Since we do a soft delete, we need to update the outbox when the label is removed + pgtrigger.Trigger( + name="issue_label_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + enriched_data JSONB; + assignee_ids JSONB; + current_label_ids JSONB; + previous_label_ids JSONB; + issue_data JSONB; + previous_issue_data JSONB; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.label.removed' + WHEN it.is_epic = true THEN 'epic.label.removed' + ELSE 'issue.label.removed' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = OLD.issue_id; + + -- If no issue found, default to 'issue.label.removed' + IF event_type_name IS NULL THEN + event_type_name := 'issue.label.removed'; + END IF; + + -- Get the complete issue data (excluding description fields) + SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped' + INTO issue_data + FROM issues i + WHERE i.id = OLD.issue_id; + + -- Fetch assignee IDs + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL; + + -- Fetch current label IDs (excluding the one being removed) + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO current_label_ids + FROM issue_labels il + WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL AND il.id != OLD.id; + + -- Fetch ALL label IDs (including the one being removed) for previous_attributes + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO previous_label_ids + FROM issue_labels il + WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL; + + -- Create enriched data with complete issue, assignee IDs, and current label IDs (after removal) + enriched_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', current_label_ids + ); + + -- Create previous attributes with complete issue, assignee IDs, and ALL label IDs (including the removed one) + previous_issue_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', previous_label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_label', + OLD.issue_id, + OLD.workspace_id, + OLD.project_id, + jsonb_build_object('data', enriched_data, 'previous_attributes', previous_issue_data), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox delete-event failed for issue_label %, reason: %', + OLD.issue_id, SQLERRM; + END; + RETURN OLD; + END; + """, + condition=None, + ), + ] + + +class IssueCommentProxy(IssueComment): + + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="issue_comment_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + enriched_data JSONB; + assignee_ids JSONB; + label_ids JSONB; + issue_data JSONB; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.comment.created' + WHEN it.is_epic = true THEN 'epic.comment.created' + ELSE 'issue.comment.created' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.comment.created' + IF event_type_name IS NULL THEN + event_type_name := 'issue.comment.created'; + END IF; + + -- Get the complete issue data (excluding description fields) + SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped' + INTO issue_data + FROM issues i + WHERE i.id = NEW.issue_id; + + -- Fetch assignee IDs + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL; + + -- Fetch label IDs + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL; + + -- Create enriched data with complete issue, assignee IDs, and label IDs + enriched_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids, + 'comment', row_to_json(NEW) + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object('data', enriched_data, 'previous_attributes', '{}'), + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for issue_comment %, reason: %', + NEW.issue_id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + pgtrigger.Trigger( + name="issue_comment_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + DECLARE + changes JSONB := '{}'; + field_name TEXT; + old_value TEXT; + new_value TEXT; + event_type_name TEXT; + enriched_data JSONB; + assignee_ids JSONB; + label_ids JSONB; + issue_data JSONB; + BEGIN + -- Check if this is a soft delete (deleted_at changed from null to not null) + IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN + -- This is a soft delete + -- Determine delete event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.comment.deleted' + WHEN it.is_epic = true THEN 'epic.comment.deleted' + ELSE 'issue.comment.deleted' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = OLD.issue_id; + + -- If no issue found, default to 'issue.comment.deleted' + IF event_type_name IS NULL THEN + event_type_name := 'issue.comment.deleted'; + END IF; + + -- Get the complete issue data (excluding description fields) + SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped' + INTO issue_data + FROM issues i + WHERE i.id = OLD.issue_id; + + -- Fetch assignee IDs + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = OLD.issue_id AND ia.deleted_at IS NULL; + + -- Fetch label IDs + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = OLD.issue_id AND il.deleted_at IS NULL; + + -- Create enriched data with complete issue, assignee IDs, and label IDs + enriched_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + OLD.issue_id, + OLD.workspace_id, + OLD.project_id, + jsonb_build_object('data', enriched_data, 'previous_attributes', row_to_json(OLD)), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox delete failed for issue_comment %, reason: %', + OLD.issue_id, SQLERRM; + END; + ELSE + -- This is a regular update, check for changes + -- Determine update event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.comment.updated' + WHEN it.is_epic = true THEN 'epic.comment.updated' + ELSE 'issue.comment.updated' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.comment.updated' + IF event_type_name IS NULL THEN + event_type_name := 'issue.comment.updated'; + END IF; + + -- Loop through all columns to detect changes + FOR field_name IN + SELECT column_name + FROM information_schema.columns + WHERE table_name = 'issue_comments' + AND table_schema = 'public' + AND column_name != 'updated_at' -- Skip updated_at column + LOOP + -- Get old and new values as text to avoid JSON conversion issues + EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) + INTO old_value, new_value + USING OLD, NEW; + + -- If values are different, add to changes + IF old_value IS DISTINCT FROM new_value THEN + changes := changes || jsonb_build_object( + field_name, + old_value + ); + END IF; + END LOOP; + + -- Only insert if there are actual changes + IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN + -- Get the complete issue data (excluding description fields) + SELECT to_jsonb(i) - 'description_html' - 'description_binary' - 'description' - 'description_stripped' + INTO issue_data + FROM issues i + WHERE i.id = NEW.issue_id; + + -- Fetch assignee IDs + SELECT COALESCE(jsonb_agg(ia.assignee_id ORDER BY ia.created_at), '[]'::jsonb) + INTO assignee_ids + FROM issue_assignees ia + WHERE ia.issue_id = NEW.issue_id AND ia.deleted_at IS NULL; + + -- Fetch label IDs + SELECT COALESCE(jsonb_agg(il.label_id ORDER BY il.created_at), '[]'::jsonb) + INTO label_ids + FROM issue_labels il + WHERE il.issue_id = NEW.issue_id AND il.deleted_at IS NULL; + + -- Create enriched data with complete issue, assignee IDs, label IDs, and updated comment + enriched_data := issue_data || jsonb_build_object( + 'assignee_ids', assignee_ids, + 'label_ids', label_ids, + 'comment', row_to_json(NEW) + ); + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object( + 'data', enriched_data, + 'previous_attributes', changes + ), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox update failed for issue_comment %, reason: %', + NEW.issue_id, SQLERRM; + END; + END IF; + END IF; + + RETURN NEW; + END; + """, + condition=None, + ), + ] + + +class IssueLinkProxy(IssueLink): + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="issue_link_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.link.added' + WHEN it.is_epic = true THEN 'epic.link.added' + ELSE 'issue.link.added' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.link.added' + IF event_type_name IS NULL THEN + event_type_name := 'issue.link.added'; + END IF; + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_link', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'), + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for issue_link %, reason: %', + NEW.issue_id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + pgtrigger.Trigger( + name="issue_link_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + DECLARE + changes JSONB := '{}'; + field_name TEXT; + old_value TEXT; + new_value TEXT; + event_type_name TEXT; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.link.updated' + WHEN it.is_epic = true THEN 'epic.link.updated' + ELSE 'issue.link.updated' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.link.updated' + IF event_type_name IS NULL THEN + event_type_name := 'issue.link.updated'; + END IF; + + -- Check if this is a soft delete (deleted_at changed from null to not null) + IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN + -- This is a soft delete + -- Determine delete event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.link.removed' + WHEN it.is_epic = true THEN 'epic.link.removed' + ELSE 'issue.link.removed' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = OLD.issue_id; + + -- If no issue found, default to 'issue.link.removed' + IF event_type_name IS NULL THEN + event_type_name := 'issue.link.removed'; + END IF; + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + OLD.issue_id, + OLD.workspace_id, + OLD.project_id, + jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox delete failed for issue %, reason: %', + OLD.id, SQLERRM; + END; + ELSE + -- This is a regular update, check for changes + -- Loop through all columns to detect changes + FOR field_name IN + SELECT column_name + FROM information_schema.columns + WHERE table_name = 'issue_links' + AND table_schema = 'public' + AND column_name != 'updated_at' -- Skip updated_at column + LOOP + -- Get old and new values as text to avoid JSON conversion issues + EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) + INTO old_value, new_value + USING OLD, NEW; + + -- If values are different, add to changes + IF old_value IS DISTINCT FROM new_value THEN + changes := changes || jsonb_build_object( + field_name, + old_value + ); + END IF; + END LOOP; + + -- Only insert if there are actual changes + IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object( + 'data', row_to_json(NEW), + 'previous_attributes', changes + ), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox update failed for issue %, reason: %', + NEW.id, SQLERRM; + END; + END IF; + END IF; + + RETURN NEW; + END; + """, + condition=None, + ), + ] + + +class IssueAttachmentProxy(FileAsset): + + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="issue_attachment_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + BEGIN + -- Only trigger for ISSUE_ATTACHMENT entity type + IF NEW.entity_type = 'ISSUE_ATTACHMENT' THEN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.attachment.added' + WHEN it.is_epic = true THEN 'epic.attachment.added' + ELSE 'issue.attachment.added' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.attachment.added' + IF event_type_name IS NULL THEN + event_type_name := 'issue.attachment.added'; + END IF; + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_attachment', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'), + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for issue_attachment %, reason: %', + NEW.issue_id, SQLERRM; + END; + END IF; + RETURN NEW; + END; + """, + condition=None, + ), + pgtrigger.Trigger( + name="issue_attachment_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + DECLARE + changes JSONB := '{}'; + field_name TEXT; + old_value TEXT; + new_value TEXT; + event_type_name TEXT; + BEGIN + -- Only trigger for ISSUE_ATTACHMENT entity type + IF NEW.entity_type = 'ISSUE_ATTACHMENT' OR OLD.entity_type = 'ISSUE_ATTACHMENT' THEN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.attachment.updated' + WHEN it.is_epic = true THEN 'epic.attachment.updated' + ELSE 'issue.attachment.updated' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.attachment.updated' + IF event_type_name IS NULL THEN + event_type_name := 'issue.attachment.updated'; + END IF; + + -- Check if this is a soft delete (deleted_at changed from null to not null) + IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN + -- This is a soft delete + -- Determine delete event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.attachment.removed' + WHEN it.is_epic = true THEN 'epic.attachment.removed' + ELSE 'issue.attachment.removed' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = OLD.issue_id; + + -- If no issue found, default to 'issue.attachment.removed' + IF event_type_name IS NULL THEN + event_type_name := 'issue.attachment.removed'; + END IF; + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_attachment', + OLD.issue_id, + OLD.workspace_id, + OLD.project_id, + jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox delete failed for issue_attachment %, reason: %', + OLD.issue_id, SQLERRM; + END; + ELSE + -- This is a regular update, check for changes + -- Loop through all columns to detect changes + FOR field_name IN + SELECT column_name + FROM information_schema.columns + WHERE table_name = 'file_assets' + AND table_schema = 'public' + AND column_name != 'updated_at' -- Skip updated_at column + LOOP + -- Get old and new values as text to avoid JSON conversion issues + EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) + INTO old_value, new_value + USING OLD, NEW; + + -- If values are different, add to changes + IF old_value IS DISTINCT FROM new_value THEN + changes := changes || jsonb_build_object( + field_name, + old_value + ); + END IF; + END LOOP; + + -- Only insert if there are actual changes + IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_attachment', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object( + 'data', row_to_json(NEW), + 'previous_attributes', changes + ), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox update failed for issue_attachment %, reason: %', + NEW.issue_id, SQLERRM; + END; + END IF; + END IF; + END IF; + + RETURN NEW; + END; + """, + condition=None, + ), + ] + + +class IssueRelationProxy(IssueRelation): + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="issue_relation_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + DECLARE + event_type_name TEXT; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.relation.added' + WHEN it.is_epic = true THEN 'epic.relation.added' + ELSE 'issue.relation.added' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.relation.added' + IF event_type_name IS NULL THEN + event_type_name := 'issue.relation.added'; + END IF; + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue_relation', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'), + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for issue_relation %, reason: %', + NEW.issue_id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + pgtrigger.Trigger( + name="issue_relation_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + DECLARE + changes JSONB := '{}'; + field_name TEXT; + old_value TEXT; + new_value TEXT; + event_type_name TEXT; + BEGIN + -- Determine event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.relation.updated' + WHEN it.is_epic = true THEN 'epic.relation.updated' + ELSE 'issue.relation.updated' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = NEW.issue_id; + + -- If no issue found, default to 'issue.relation.updated' + IF event_type_name IS NULL THEN + event_type_name := 'issue.relation.updated'; + END IF; + + -- Check if this is a soft delete (deleted_at changed from null to not null) + IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN + -- This is a soft delete + -- Determine delete event type based on issue type + SELECT + CASE + WHEN i.type_id IS NULL THEN 'issue.relation.removed' + WHEN it.is_epic = true THEN 'epic.relation.removed' + ELSE 'issue.relation.removed' + END + INTO event_type_name + FROM issues i + LEFT JOIN issue_types it ON it.id = i.type_id + WHERE i.id = OLD.issue_id; + + -- If no issue found, default to 'issue.relation.removed' + IF event_type_name IS NULL THEN + event_type_name := 'issue.relation.removed'; + END IF; + + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + OLD.issue_id, + OLD.workspace_id, + OLD.project_id, + jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox delete failed for issue_relation %, reason: %', + OLD.issue_id, SQLERRM; + END; + ELSE + -- This is a regular update, check for changes + -- Loop through all columns to detect changes + FOR field_name IN + SELECT column_name + FROM information_schema.columns + WHERE table_name = 'issue_relations' + AND table_schema = 'public' + AND column_name != 'updated_at' -- Skip updated_at column + LOOP + -- Get old and new values as text to avoid JSON conversion issues + EXECUTE format('SELECT ($1).%I::text, ($2).%I::text', field_name, field_name) + INTO old_value, new_value + USING OLD, NEW; + + -- If values are different, add to changes + IF old_value IS DISTINCT FROM new_value THEN + changes := changes || jsonb_build_object( + field_name, + old_value + ); + END IF; + END LOOP; + + -- Only insert if there are actual changes + IF jsonb_typeof(changes) = 'object' AND changes != '{}' THEN + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, workspace_id, project_id, payload, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + event_type_name, + 'issue', + NEW.issue_id, + NEW.workspace_id, + NEW.project_id, + jsonb_build_object( + 'data', row_to_json(NEW), + 'previous_attributes', changes + ), + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox update failed for issue_relation %, reason: %', + NEW.issue_id, SQLERRM; + END; + END IF; + END IF; + RETURN NEW; + END; + """, + condition=None, + ), + ] diff --git a/apps/api/plane/event_stream/models/module.py b/apps/api/plane/event_stream/models/module.py new file mode 100644 index 0000000000..7c4e766a34 --- /dev/null +++ b/apps/api/plane/event_stream/models/module.py @@ -0,0 +1,92 @@ +import pgtrigger +from plane.db.models import ModuleIssue + + +class ModuleIssueProxy(ModuleIssue): + class Meta: + proxy = True + triggers = [ + pgtrigger.Trigger( + name="module_issue_outbox_insert", + operation=pgtrigger.Insert, + when=pgtrigger.After, + func=""" + BEGIN + BEGIN + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + 'issue.module.added', + 'module_issue', + NEW.issue_id, + jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', '{}'), + NEW.workspace_id, + NEW.project_id, + now(), + NEW.created_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox insert failed for module_issue %, reason: %', + NEW.issue_id, SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + # Handle both soft deletes (deleted_at updated to not null) and regular updates + pgtrigger.Trigger( + name="module_issue_outbox_update", + operation=pgtrigger.Update, + when=pgtrigger.After, + func=""" + BEGIN + BEGIN + -- Check if this is a soft delete (deleted_at changed from null to not null) + IF OLD.deleted_at IS NULL AND NEW.deleted_at IS NOT NULL THEN + -- This is a soft delete + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + 'issue.module.removed', + 'module_issue', + OLD.issue_id, + jsonb_build_object('data', '{}', 'previous_attributes', row_to_json(OLD)), + OLD.workspace_id, + OLD.project_id, + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + ELSE + -- This is a regular update + INSERT INTO outbox (event_id, event_type, entity_type, entity_id, payload, workspace_id, project_id, created_at, initiator_id, initiator_type) + VALUES ( + gen_random_uuid(), + 'issue.module.moved', + 'module_issue', + NEW.issue_id, + jsonb_build_object('data', row_to_json(NEW), 'previous_attributes', row_to_json(OLD)), + NEW.workspace_id, + NEW.project_id, + now(), + NEW.updated_by_id, + COALESCE(current_setting('plane.initiator_type', true), 'USER') + ) + ON CONFLICT DO NOTHING; + END IF; + EXCEPTION + WHEN others THEN + RAISE WARNING 'Outbox update-event failed for module_issue %, reason: %', + COALESCE(NEW.issue_id, OLD.issue_id), SQLERRM; + END; + RETURN NEW; + END; + """, + condition=None, + ), + ] diff --git a/apps/api/plane/event_stream/models/outbox.py b/apps/api/plane/event_stream/models/outbox.py new file mode 100644 index 0000000000..5c89bd278e --- /dev/null +++ b/apps/api/plane/event_stream/models/outbox.py @@ -0,0 +1,267 @@ +# Python +from uuid import uuid4, UUID +from dataclasses import dataclass, asdict +from typing import Union, Dict, Any, Optional +from datetime import datetime +import json + +# Django +from django.db import models +from django.utils import timezone + + +class InitiatorTypes(models.TextChoices): + USER = "USER" + SYSTEM_IMPORT = "SYSTEM.IMPORT" + SYSTEM_AUTOMATION = "SYSTEM.AUTOMATION" + + +class Outbox(models.Model): + # Primary identifier + id = models.BigAutoField(primary_key=True) + event_id = models.UUIDField(default=uuid4, editable=False, unique=True) + + # What event occurred (e.g. "issue.created", "issue.updated", "issue.deleted") + event_type = models.CharField(max_length=255) + + # entity type -- like issue, project, etc. + entity_type = models.CharField(max_length=255) + # the id of the entity that was affected by the event + entity_id = models.UUIDField() + + # The actual payload to send (JSON structure) + payload = models.JSONField() + + # The date and time the event was processed (if it was processed) + processed_at = models.DateTimeField(null=True, blank=True) + + # Audit fields + created_at = models.DateTimeField(default=timezone.now) + + # The date and time the event was claimed (if it was claimed) + claimed_at = models.DateTimeField(null=True, blank=True) + + # The workspace ID that the event belongs to + workspace_id = models.UUIDField() + # The project ID that the event belongs to + project_id = models.UUIDField() + + # The user ID that the event belongs to + initiator_id = models.UUIDField(help_text="The user ID who triggered the event") + + # The type of initiator that triggered the event + initiator_type = models.CharField(max_length=255, default=InitiatorTypes.USER) + + class Meta: + db_table = "outbox" + indexes = [ + # live queue β†’ tiny, always‑cached + models.Index( + fields=["claimed_at", "processed_at", "id"], + name="outbox_unclaimed_unprocessed", + condition=models.Q(claimed_at__isnull=True, processed_at__isnull=True), + ), + # analytics dashboards + models.Index( + fields=["processed_at"], + name="outbox_processed_idx", + condition=models.Q(processed_at__isnull=False), + ), + ] + ordering = ["-created_at"] + + def __str__(self): + return ( + f"Outbox<{self.event_type}:{self.entity_type}:{self.entity_id}> {self.id}" + ) + + +@dataclass +class OutboxEvent: + """ + Represents an event from the outbox table. + + This dataclass provides a type-safe representation of outbox events + that can be used consistently across the publisher and poller components. + """ + + # Database fields + id: int + event_id: Union[UUID, str] + event_type: str + entity_type: str + entity_id: Union[UUID, str] + payload: Dict[str, Any] + created_at: datetime + workspace_id: Union[UUID, str] + project_id: Union[UUID, str] + initiator_id: Union[UUID, str] + initiator_type: str + + # Optional fields + processed_at: Optional[datetime] = None + claimed_at: Optional[datetime] = None + + @classmethod + def from_db_row(cls, row: tuple) -> "OutboxEvent": + """ + Create an OutboxEvent from a database row tuple. + + Expected row format from database query: + (id, event_id, event_type, entity_type, entity_id, payload, processed_at, created_at, claimed_at) + + Args: + row: Tuple containing database row data + + Returns: + OutboxEvent instance + """ + ( + id, + event_id, + event_type, + entity_type, + entity_id, + payload, + processed_at, + created_at, + claimed_at, + workspace_id, + project_id, + initiator_id, + initiator_type, + ) = row + + return cls( + id=id, + event_id=event_id, + event_type=event_type, + entity_type=entity_type, + entity_id=str(entity_id), # Convert UUID to string for consistency + payload=payload, + processed_at=processed_at, + created_at=created_at, + claimed_at=claimed_at, + workspace_id=workspace_id, + project_id=project_id, + initiator_id=initiator_id, + initiator_type=initiator_type, + ) + + @classmethod + def from_dict(cls, data: Dict[str, Any]) -> "OutboxEvent": + """ + Create an OutboxEvent from a dictionary. + + Args: + data: Dictionary containing event data + + Returns: + OutboxEvent instance + """ + return cls( + id=data["id"], + event_id=data["event_id"], + event_type=data["event_type"], + entity_type=data["entity_type"], + entity_id=str(data["entity_id"]), + payload=data["payload"], + processed_at=data.get("processed_at"), + created_at=data["created_at"], + claimed_at=data.get("claimed_at"), + workspace_id=data["workspace_id"], + project_id=data["project_id"], + initiator_id=data["initiator_id"], + initiator_type=data["initiator_type"], + ) + + def to_dict(self) -> Dict[str, Any]: + """ + Convert the OutboxEvent to a dictionary. + + Returns: + Dictionary representation of the event + """ + return asdict(self) + + def to_publisher_format(self) -> Dict[str, Any]: + """ + Convert to the format expected by the event stream publisher. + + Returns: + Dictionary formatted for publishing + """ + return { + "event_id": str(self.event_id), + "event_type": self.event_type, + "entity_type": self.entity_type, + "entity_id": str(self.entity_id), + "payload": self.payload, + "workspace_id": str(self.workspace_id), + "project_id": str(self.project_id), + "initiator_id": str(self.initiator_id), + "initiator_type": self.initiator_type, + } + + def to_json(self) -> str: + """ + Convert the OutboxEvent to a JSON string. + + Returns: + JSON string representation + """ + + def json_serializer(obj): + """Custom JSON serializer for datetime and UUID objects.""" + if isinstance(obj, datetime): + return obj.isoformat() + elif isinstance(obj, UUID): + return str(obj) + raise TypeError( + f"Object of type {type(obj).__name__} is not JSON serializable" + ) + + return json.dumps(self.to_dict(), default=json_serializer) + + def get_metadata(self) -> Dict[str, Any]: + """ + Get metadata about this event for logging/monitoring. + + Returns: + Dictionary containing event metadata + """ + return { + "source": "outbox-poller", + "outbox_id": self.id, + "event_id": str(self.event_id), + "event_type": self.event_type, + "entity_type": self.entity_type, + "entity_id": str(self.entity_id), + "created_at": ( + self.created_at.isoformat() + if isinstance(self.created_at, datetime) + else self.created_at + ), + "workspace_id": str(self.workspace_id), + "project_id": str(self.project_id), + "initiator_id": str(self.initiator_id), + "initiator_type": self.initiator_type, + } + + def __str__(self) -> str: + return f"OutboxEvent<{self.event_type}:{self.entity_type}:{self.entity_id}> {self.id}" + + def __repr__(self) -> str: + return ( + f"OutboxEvent(" + f"id={self.id}, " + f"event_id='{self.event_id}', " + f"event_type='{self.event_type}', " + f"entity_type='{self.entity_type}', " + f"entity_id='{self.entity_id}', " + f"workspace_id='{self.workspace_id}', " + f"project_id='{self.project_id}', " + f"initiator_id='{self.initiator_id}', " + f"initiator_type='{self.initiator_type}'" + f")" + ) diff --git a/apps/api/plane/payment/flags/provider.py b/apps/api/plane/payment/flags/provider.py index 86dc001e05..311d09375d 100644 --- a/apps/api/plane/payment/flags/provider.py +++ b/apps/api/plane/payment/flags/provider.py @@ -5,9 +5,7 @@ import requests from django.conf import settings # Third party imports -from openfeature.provider import AbstractProvider -from openfeature.provider.metadata import Metadata -from openfeature.flag_evaluation import FlagResolutionDetails +from openfeature.provider import AbstractProvider, FlagResolutionDetails, Metadata # Module imports from plane.utils.exception_logger import log_exception diff --git a/apps/api/plane/settings/common.py b/apps/api/plane/settings/common.py index 53806ba602..ead345f3c4 100644 --- a/apps/api/plane/settings/common.py +++ b/apps/api/plane/settings/common.py @@ -53,12 +53,14 @@ INSTALLED_APPS = [ "plane.graphql", "plane.payment", "plane.silo", + "plane.event_stream", # Third-party things "strawberry.django", "rest_framework", "oauth2_provider", "corsheaders", "django_celery_beat", + "pgtrigger", ] # Middlewares diff --git a/apps/api/plane/settings/local.py b/apps/api/plane/settings/local.py index 07d35f5a6a..76e3165cbb 100644 --- a/apps/api/plane/settings/local.py +++ b/apps/api/plane/settings/local.py @@ -44,7 +44,7 @@ LOGGING = { "style": "{", }, "json": { - "()": "pythonjsonlogger.jsonlogger.JsonFormatter", + "()": "pythonjsonlogger.json.JsonFormatter", "fmt": "%(levelname)s %(asctime)s %(module)s %(name)s %(message)s", }, }, @@ -73,5 +73,10 @@ LOGGING = { "handlers": ["console"], "propagate": False, }, + "plane.event_stream": { + "level": "INFO", + "handlers": ["console"], + "propagate": False, + }, }, } diff --git a/apps/api/plane/settings/mongo.py b/apps/api/plane/settings/mongo.py new file mode 100644 index 0000000000..ad90ca3d7e --- /dev/null +++ b/apps/api/plane/settings/mongo.py @@ -0,0 +1,118 @@ +# Django imports +from django.conf import settings +import logging + +# Third party imports +from pymongo import MongoClient +from pymongo.database import Database +from pymongo.collection import Collection +from typing import Optional, TypeVar, Type + + +T = TypeVar("T", bound="MongoConnection") + +# Set up logger +logger = logging.getLogger(__name__) + + +class MongoConnection: + """ + A singleton class that manages MongoDB connections. + + This class ensures only one MongoDB connection is maintained throughout the application. + It provides methods to access the MongoDB client, database, and collections. + + Attributes: + _instance (Optional[MongoConnection]): The singleton instance of this class + _client (Optional[MongoClient]): The MongoDB client instance + _db (Optional[Database]): The MongoDB database instance + """ + + _instance: Optional["MongoConnection"] = None + _client: Optional[MongoClient] = None + _db: Optional[Database] = None + + def __new__(cls: Type[T]) -> T: + """ + Creates a new instance of MongoConnection if one doesn't exist. + + Returns: + MongoConnection: The singleton instance + """ + if cls._instance is None: + cls._instance = super(MongoConnection, cls).__new__(cls) + try: + if not settings.MONGO_DB_URL or not settings.MONGO_DB_DATABASE: + logger.warning( + "MongoDB connection parameters not configured. MongoDB functionality will be disabled." + ) + return cls._instance + + cls._client = MongoClient(settings.MONGO_DB_URL) + cls._db = cls._client[settings.MONGO_DB_DATABASE] + + # Test the connection + cls._client.server_info() + logger.info("MongoDB connection established successfully") + except Exception as e: + logger.warning( + f"Failed to initialize MongoDB connection: {str(e)}. MongoDB functionality will be disabled." + ) + return cls._instance + + @classmethod + def get_client(cls) -> Optional[MongoClient]: + """ + Returns the MongoDB client instance. + + Returns: + Optional[MongoClient]: The MongoDB client instance or None if not configured + """ + if cls._client is None: + cls._instance = cls() + return cls._client + + @classmethod + def get_db(cls) -> Optional[Database]: + """ + Returns the MongoDB database instance. + + Returns: + Optional[Database]: The MongoDB database instance or None if not configured + """ + if cls._db is None: + cls._instance = cls() + return cls._db + + @classmethod + def get_collection(cls, collection_name: str) -> Optional[Collection]: + """ + Returns a MongoDB collection by name. + + Args: + collection_name (str): The name of the collection to retrieve + + Returns: + Optional[Collection]: The MongoDB collection instance or None if not configured + """ + try: + db = cls.get_db() + if db is None: + logger.warning( + f"Cannot access collection '{collection_name}': MongoDB not configured" + ) + return None + return db[collection_name] + except Exception as e: + logger.warning(f"Failed to access collection '{collection_name}': {str(e)}") + return None + + @classmethod + def is_configured(cls) -> bool: + """ + Check if MongoDB is properly configured and connected. + + Returns: + bool: True if MongoDB is configured and connected, False otherwise + """ + return cls._client is not None and cls._db is not None diff --git a/apps/api/plane/settings/production.py b/apps/api/plane/settings/production.py index e68876d4a7..f3f9b40b84 100644 --- a/apps/api/plane/settings/production.py +++ b/apps/api/plane/settings/production.py @@ -10,8 +10,6 @@ DEBUG = int(os.environ.get("DEBUG", 0)) == 1 # Honor the 'X-Forwarded-Proto' header for request.is_secure() SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https") -INSTALLED_APPS += ("scout_apm.django",) # noqa - if os.environ.get("IS_MULTI_CLOUD", "0") == "1": SECURE_SSL_REDIRECT = True else: @@ -92,6 +90,11 @@ LOGGING = { "handlers": ["console"], "propagate": False, }, + "plane.event_stream": { + "level": "INFO", + "handlers": ["console"], + "propagate": False, + }, }, } diff --git a/apps/api/plane/silo/bgtasks/bulk_update_issue_relations_task.py b/apps/api/plane/silo/bgtasks/bulk_update_issue_relations_task.py index f9ad6abd9a..58eae522f0 100644 --- a/apps/api/plane/silo/bgtasks/bulk_update_issue_relations_task.py +++ b/apps/api/plane/silo/bgtasks/bulk_update_issue_relations_task.py @@ -3,6 +3,7 @@ from celery import shared_task from plane.db.models import Issue, IssueRelation from plane.ee.models.job import ImportJob +from django.db import transaction, connection logger = logging.getLogger("plane.worker") @@ -70,9 +71,11 @@ def bulk_update_issue_relations_task(job_id: str, user_id: str | None = None): created_by_id=user_id, ) ) - - # Bulk create relations, ignoring any duplicates - IssueRelation.objects.bulk_create(issue_relations, ignore_conflicts=True) + with transaction.atomic(): + with connection.cursor() as cur: + cur.execute("SET LOCAL plane.initiator_type = 'SYSTEM.IMPORT'") + # Bulk create relations, ignoring any duplicates + IssueRelation.objects.bulk_create(issue_relations, ignore_conflicts=True) # Handle parent_id relations for ex_issue_id, parent_ex_id in parent_id_relations: @@ -83,7 +86,9 @@ def bulk_update_issue_relations_task(job_id: str, user_id: str | None = None): logger.warning(f"Issue {ex_issue_id} or parent {parent_ex_id} not found") continue - # Update parent_id - Issue.objects.filter(id=issue_id).update( - parent_id=parent_id, updated_by_id=user_id - ) + with connection.cursor() as cur: + cur.execute("SET LOCAL plane.initiator_type = 'SYSTEM.IMPORT'") + # Update parent_id + Issue.objects.filter(id=issue_id).update( + parent_id=parent_id, updated_by_id=user_id + ) diff --git a/apps/api/plane/tests/conftest.py b/apps/api/plane/tests/conftest.py index f7ad5a9eaa..266dddaa26 100644 --- a/apps/api/plane/tests/conftest.py +++ b/apps/api/plane/tests/conftest.py @@ -58,7 +58,7 @@ def workspace(db, create_user): def project(db, workspace, create_user): """Create and return a project instance for OAuth testing""" from plane.tests.factories import ProjectFactory - + return ProjectFactory( workspace=workspace, created_by=create_user, @@ -193,20 +193,20 @@ def plane_server(live_server): def oauth_application(db, create_user, workspace): """Create and return an OAuth application instance""" from plane.tests.factories import ApplicationFactory, ApplicationOwnerFactory - + app = ApplicationFactory( user=create_user, created_by=create_user, updated_by=create_user, ) - + # Create application owner ApplicationOwnerFactory( user=create_user, application=app, workspace=workspace, ) - + return app @@ -215,7 +215,7 @@ def workspace_app_installation(db, workspace, oauth_application, create_user): """Create and return a workspace app installation instance""" from plane.tests.factories import WorkspaceAppInstallationFactory from plane.authentication.models import WorkspaceAppInstallation - + # When this factory creates and saves the WorkspaceAppInstallation, # the model's save() method automatically creates a bot user and adds it # to workspace and project members @@ -226,6 +226,7 @@ def workspace_app_installation(db, workspace, oauth_application, create_user): status=WorkspaceAppInstallation.Status.INSTALLED, ) + @pytest.fixture def workspace(create_user): """ diff --git a/apps/api/plane/tests/contract/app/test_issue_duplicate.py b/apps/api/plane/tests/contract/app/test_issue_duplicate.py index 27fe640ecb..b46bfeb614 100644 --- a/apps/api/plane/tests/contract/app/test_issue_duplicate.py +++ b/apps/api/plane/tests/contract/app/test_issue_duplicate.py @@ -14,7 +14,6 @@ from unittest.mock import MagicMock @pytest.mark.contract class TestIssueDuplicateEndpointContrasts: - @pytest.mark.django_db @pytest.fixture def base_setup(self, session_client, create_user): """Base setup for all tests""" diff --git a/apps/api/plane/tests/unit/bg_tasks/test_outbox_cleaner.py b/apps/api/plane/tests/unit/bg_tasks/test_outbox_cleaner.py new file mode 100644 index 0000000000..79796718e6 --- /dev/null +++ b/apps/api/plane/tests/unit/bg_tasks/test_outbox_cleaner.py @@ -0,0 +1,226 @@ +import uuid +from datetime import timedelta +from unittest.mock import patch, MagicMock +import pytest +from django.utils import timezone +from pymongo.errors import BulkWriteError +from pymongo.operations import InsertOne + +from plane.event_stream.models.outbox import Outbox +from plane.event_stream.bgtasks.outbox_cleaner import ( + flush_to_mongo_and_delete, + delete_outbox_records, +) + + +@pytest.fixture +def mock_mongo_collection(): + """Create a mock MongoDB collection""" + return MagicMock() + + +@pytest.fixture +def old_outbox_records(db, workspace, project): + """Create old outbox records that should be processed""" + cutoff_time = timezone.now() - timedelta(days=2) + records = [] + + for i in range(3): + record = Outbox.objects.create( + event_type=f"issue.old.{i}", + entity_type="issue", + entity_id=uuid.uuid4(), + payload={"issue_id": f"old-{i}"}, + processed_at=cutoff_time - timedelta(hours=1), + workspace_id=workspace.id, + project_id=project.id, + ) + + records.append(record) + + return records + + +@pytest.fixture +def recent_outbox_records(db, workspace, project): + """Create recent outbox records that should not be processed""" + records = [] + + for i in range(2): + record = Outbox.objects.create( + event_type=f"issue.recent.{i}", + entity_type="issue", + entity_id=uuid.uuid4(), + payload={"issue_id": f"recent-{i}"}, + processed_at=timezone.now() - timedelta(hours=1), + workspace_id=workspace.id, + project_id=project.id, + ) + records.append(record) + + return records + + +@pytest.mark.unit +@pytest.mark.django_db +class TestFlushToMongoAndDelete: + """Test the flush_to_mongo_and_delete function""" + + def test_flush_to_mongo_and_delete_success( + self, mock_mongo_collection, workspace, project + ): + """Test successful MongoDB insert and PostgreSQL deletion""" + # Arrange + buffer = [ + { + "event_id": str(uuid.uuid4()), + "event_type": "issue.created", + "entity_type": "issue", + "entity_id": str(uuid.uuid4()), + "payload": {"issue_id": "123"}, + "processed_at": timezone.now() - timedelta(days=3), + "created_at": timezone.now() - timedelta(days=3), + "workspace_id": str(uuid.uuid4()), + "project_id": str(uuid.uuid4()), + } + ] + ids_to_delete = [1] + + # Act + flush_to_mongo_and_delete(mock_mongo_collection, buffer, ids_to_delete) + + # Assert + expected_operations = [InsertOne(doc) for doc in buffer] + mock_mongo_collection.bulk_write.assert_called_once_with(expected_operations) + + def test_flush_to_mongo_and_delete_empty_buffer(self, mock_mongo_collection): + """Test handling of empty buffer""" + # Act + flush_to_mongo_and_delete(mock_mongo_collection, [], []) + + # Assert + mock_mongo_collection.bulk_write.assert_not_called() + + def test_flush_to_mongo_and_delete_mongodb_error(self, mock_mongo_collection): + """Test handling of MongoDB bulk write error""" + # Arrange + buffer = [{"event_id": str(uuid.uuid4()), "event_type": "issue.created"}] + mock_mongo_collection.bulk_write.side_effect = BulkWriteError( + results=MagicMock() + ) + + # Act & Assert - should not raise exception + flush_to_mongo_and_delete(mock_mongo_collection, buffer, [1]) + + # Verify MongoDB was still called + mock_mongo_collection.bulk_write.assert_called_once() + + def test_flush_to_mongo_and_delete_with_real_records( + self, db, mock_mongo_collection, workspace, project + ): + """Test with actual Outbox records in database""" + # Arrange + outbox = Outbox.objects.create( + event_type="issue.created", + entity_type="issue", + entity_id=uuid.uuid4(), + payload={"issue_id": "123"}, + processed_at=timezone.now() - timedelta(days=3), + workspace_id=workspace.id, + project_id=project.id, + ) + + buffer = [ + { + "event_id": str(outbox.event_id), + "event_type": outbox.event_type, + "entity_type": outbox.entity_type, + "entity_id": str(outbox.entity_id), + "payload": outbox.payload, + "processed_at": outbox.processed_at, + "created_at": outbox.created_at, + "workspace_id": str(outbox.workspace_id), + "project_id": str(outbox.project_id), + } + ] + ids_to_delete = [outbox.id] + + # Act + flush_to_mongo_and_delete(mock_mongo_collection, buffer, ids_to_delete) + + # Assert + mock_mongo_collection.bulk_write.assert_called_once() + assert not Outbox.objects.filter(id=outbox.id).exists() + + +@pytest.mark.unit +@pytest.mark.django_db +class TestDeleteOutboxRecords: + """Test the delete_outbox_records Celery task""" + + @patch("plane.event_stream.bgtasks.outbox_cleaner.MongoConnection") + def test_delete_outbox_records_mongodb_available( + self, mock_mongo_connection, old_outbox_records, recent_outbox_records + ): + """Test outbox cleanup with MongoDB available""" + # Arrange + mock_mongo_connection.is_configured.return_value = True + mock_collection = MagicMock() + mock_mongo_connection.get_collection.return_value = mock_collection + + # Act + delete_outbox_records() + + # Assert + # Check that old records were processed (deleted from PostgreSQL) + for record in old_outbox_records: + assert not Outbox.objects.filter(id=record.id).exists() + + # Check that recent records were not processed + for record in recent_outbox_records: + assert Outbox.objects.filter(id=record.id).exists() + + # Check MongoDB was called + mock_mongo_connection.is_configured.assert_called_once() + mock_mongo_connection.get_collection.assert_called_once_with("outbox") + + @patch("plane.event_stream.bgtasks.outbox_cleaner.MongoConnection") + def test_delete_outbox_records_mongodb_unavailable( + self, mock_mongo_connection, old_outbox_records, recent_outbox_records + ): + """Test outbox cleanup when MongoDB is not available""" + # Arrange + mock_mongo_connection.is_configured.return_value = False + + # Act + delete_outbox_records() + + # Assert + # Check that old records were still deleted from PostgreSQL + for record in old_outbox_records: + assert not Outbox.objects.filter(id=record.id).exists() + + # Check that recent records were not processed + for record in recent_outbox_records: + assert Outbox.objects.filter(id=record.id).exists() + + # Check MongoDB was checked but not used + mock_mongo_connection.is_configured.assert_called_once() + mock_mongo_connection.get_collection.assert_not_called() + + @patch("plane.event_stream.bgtasks.outbox_cleaner.MongoConnection") + def test_delete_outbox_records_no_records_to_process(self, mock_mongo_connection): + """Test outbox cleanup when no records need processing""" + # Arrange + Outbox.objects.all().delete() + mock_mongo_connection.is_configured.return_value = True + mock_collection = MagicMock() + mock_mongo_connection.get_collection.return_value = mock_collection + + # Act + delete_outbox_records() + + # Assert + mock_mongo_connection.is_configured.assert_called_once() + mock_mongo_connection.get_collection.assert_called_once() + mock_collection.bulk_write.assert_not_called() diff --git a/apps/api/plane/tests/unit/commands/__init__.py b/apps/api/plane/tests/unit/commands/__init__.py new file mode 100644 index 0000000000..e69de29bb2 diff --git a/apps/api/plane/tests/unit/commands/test_outbox_poller.py b/apps/api/plane/tests/unit/commands/test_outbox_poller.py new file mode 100644 index 0000000000..f6a1a80036 --- /dev/null +++ b/apps/api/plane/tests/unit/commands/test_outbox_poller.py @@ -0,0 +1,536 @@ +import asyncio +from unittest.mock import AsyncMock, MagicMock, patch +from uuid import uuid4, UUID +import pytest +from django.utils import timezone +from datetime import datetime, timedelta + +from plane.event_stream.models.outbox import Outbox +from plane.event_stream.management.commands.outbox_poller import ( + OutboxPoller, + DatabaseConnectionPool, + MemoryMonitor, +) + + +@pytest.fixture +def outbox_poller(): + """Create an OutboxPoller instance with test configuration""" + return OutboxPoller( + batch_size=10, + interval_min=0.1, + interval_max=1.0, + memory_limit_mb=100, + memory_check_interval=5, + ) + + +def create_outbox_records( + count: int = 3, + workspace_id: UUID = None, + project_id: UUID = None, + processed_at: datetime = None, +): + """Create outbox records for testing""" + records = [] + for i in range(count): + record = Outbox.objects.create( + event_type=f"issue.updated.{i}", + entity_type="issue", + entity_id=uuid4(), + payload={ + "issue_id": f"issue-{i}", + "status": "completed" if processed_at else "pending", + }, + processed_at=processed_at, + workspace_id=workspace_id, + project_id=project_id, + ) + records.append(record) + return records + + +def delete_outbox_records(processed: bool = False) -> None: + """Delete all outbox records""" + Outbox.objects.filter(processed_at__isnull=processed).delete() + + +@pytest.fixture +def mock_handler(): + """Create a mock handler function for testing""" + mock = MagicMock() + # Set a name for the mock to avoid AttributeError + mock.__name__ = "mock_handler" + return mock + + +@pytest.mark.unit +class TestOutboxPoller: + """Test the OutboxPoller class""" + + def test_outbox_poller_initialization(self, outbox_poller): + """Test OutboxPoller is initialized with correct parameters""" + assert outbox_poller.batch_size == 10 + assert outbox_poller.interval_min == 0.1 + assert outbox_poller.interval_max == 1.0 + assert outbox_poller.memory_limit_mb == 100 + assert outbox_poller.memory_check_interval == 5 + assert outbox_poller.handlers == [] + + def test_add_handler(self, outbox_poller, mock_handler): + """Test adding handlers to the poller""" + outbox_poller.add_handler(mock_handler) + assert len(outbox_poller.handlers) == 1 + assert outbox_poller.handlers[0] == mock_handler + + @pytest.mark.asyncio + async def test_process_event_success( + self, outbox_poller, mock_handler, workspace, project + ): + """Test successful event processing""" + outbox_poller.add_handler(mock_handler) + + # Create a mock row tuple + row = ( + 1, # id + uuid4(), # event_id + "issue.created", # event_type + "issue", # entity_type + uuid4(), # entity_id + {"issue_id": "123", "title": "Test Issue"}, # payload + None, # processed_at + timezone.now(), # created_at + None, # claimed_at + workspace.id, # workspace_id + project.id, + ) + + result = await outbox_poller._process_event(row) + + assert result is True + mock_handler.assert_called_once() + + # Verify the event data passed to handler + call_args = mock_handler.call_args[0][0] + assert call_args.id == 1 + assert call_args.event_type == "issue.created" + assert call_args.entity_type == "issue" + assert call_args.payload == {"issue_id": "123", "title": "Test Issue"} + assert call_args.workspace_id == workspace.id + assert call_args.project_id == project.id + + @pytest.mark.asyncio + async def test_process_event_handler_error( + self, outbox_poller, mock_handler, workspace, project + ): + """Test event processing when handler raises an exception""" + # Make the handler raise an exception + mock_handler.side_effect = Exception("Handler error") + outbox_poller.add_handler(mock_handler) + + row = ( + 1, + uuid4(), + "issue.created", + "issue", + uuid4(), + {"issue_id": "123"}, + None, + timezone.now(), + None, + workspace.id, + project.id, + ) + + result = await outbox_poller._process_event(row) + + assert result is False + mock_handler.assert_called_once() + + @pytest.mark.asyncio + async def test_process_event_lambda_handler_error( + self, outbox_poller, workspace, project + ): + """Test event processing when lambda handler raises an exception""" + # Lambda function that raises an exception + lambda_handler = lambda x: (_ for _ in ()).throw(Exception("Lambda error")) + outbox_poller.add_handler(lambda_handler) + + row = ( + 1, + uuid4(), + "issue.created", + "issue", + uuid4(), + {"issue_id": "123"}, + None, + timezone.now(), + None, + workspace.id, + project.id, + ) + + result = await outbox_poller._process_event(row) + + assert result is False + + @pytest.mark.asyncio + async def test_process_event_async_handler(self, outbox_poller, workspace, project): + """Test processing event with async handler""" + + async def async_handler(event_data): + await asyncio.sleep(0.01) # Simulate async work + return True + + outbox_poller.add_handler(async_handler) + + row = ( + 1, + uuid4(), + "issue.created", + "issue", + uuid4(), + {"issue_id": "123"}, + None, + timezone.now(), + None, + workspace.id, + project.id, + ) + + result = await outbox_poller._process_event(row) + assert result is True + + @pytest.mark.asyncio + async def test_process_event_multiple_handlers( + self, outbox_poller, workspace, project + ): + """Test processing event with multiple handlers""" + handler1 = MagicMock() + handler1.__name__ = "handler1" + handler2 = MagicMock() + handler2.__name__ = "handler2" + + outbox_poller.add_handler(handler1) + outbox_poller.add_handler(handler2) + + row = ( + 1, + uuid4(), + "issue.created", + "issue", + uuid4(), + {"issue_id": "123"}, + None, + timezone.now(), + None, + workspace.id, + project.id, + ) + + result = await outbox_poller._process_event(row) + + assert result is True + handler1.assert_called_once() + handler2.assert_called_once() + + @pytest.mark.asyncio + async def test_process_event_mixed_handler_types( + self, outbox_poller, mock_handler, workspace, project + ): + """Test processing event with mixed handler types (normal, lambda, mock)""" + + def normal_handler(event_data): + return "normal" + + outbox_poller.add_handler(normal_handler) + outbox_poller.add_handler(mock_handler) + + row = ( + 1, + uuid4(), + "issue.created", + "issue", + uuid4(), + {"issue_id": "123"}, + None, + timezone.now(), + None, + workspace.id, + project.id, + ) + + result = await outbox_poller._process_event(row) + assert result is True + + +@pytest.mark.unit +@pytest.mark.django_db +class TestDatabaseConnectionPool: + """Test the DatabaseConnectionPool class""" + + @pytest.mark.asyncio + async def test_fetch_and_lock_rows_with_real_connection(self, workspace, project): + """Test fetch_and_lock_rows with real database connection using fixtures""" + delete_outbox_records(processed=False) + + # Create unprocessed outbox records + create_outbox_records( + processed_at=None, count=3, workspace_id=workspace.id, project_id=project.id + ) + + async with DatabaseConnectionPool() as db_pool: + # Fetch unprocessed records using the real connection + rows = await db_pool.fetch_and_lock_rows(10) + + # Should fetch exactly 3 unprocessed records from our fixture + assert len(rows) == 3 + + # Verify the structure of returned rows + for i, row in enumerate(rows): + assert ( + len(row) == 11 + ) # id, event_id, event_type, entity_type, entity_id, payload, processed_at, created_at, claimed_at, workspace_id, project_id + assert row[0] is not None # id + assert row[1] is not None # event_id + assert row[2] == f"issue.updated.{i}" # event_type + assert row[3] == "issue" # entity_type + assert row[4] is not None # entity_id + assert row[5] == { + "issue_id": f"issue-{i}", + "status": "pending", + } # payload + assert ( + row[6] is None + ) # processed_at should be None for unprocessed records + assert row[7] is not None # created_at + assert row[9] == workspace.id # workspace_id + assert row[10] == project.id # project_id + # Delete the records + delete_outbox_records(processed=False) + + # Delete the records + delete_outbox_records(processed=False) + + @pytest.mark.asyncio + async def test_mark_processed_with_real_connection(self, workspace, project): + """Test mark_processed with real database connection using fixtures""" + delete_outbox_records(processed=False) + + # Create unprocessed outbox records + create_outbox_records( + processed_at=None, count=3, workspace_id=workspace.id, project_id=project.id + ) + + # Get the IDs of unprocessed records + record_ids = [record.id for record in Outbox.objects.all()] + + async with DatabaseConnectionPool() as db_pool: + # Mark records as processed using the real connection + result = await db_pool.mark_processed(record_ids) + + # Should successfully mark all records as processed + assert result is True + + # Verify records are now marked as processed in the database + for record_id in record_ids: + record = Outbox.objects.get(id=record_id) + assert record.processed_at is not None + + # Delete the records + delete_outbox_records(processed=False) + + @pytest.mark.asyncio + async def test_fetch_and_lock_rows_batch_size_limit(self, workspace, project): + """Test that fetch_and_lock_rows respects batch size limits""" + delete_outbox_records(processed=False) + + # Create unprocessed outbox records + create_outbox_records( + processed_at=None, count=3, workspace_id=workspace.id, project_id=project.id + ) + + async with DatabaseConnectionPool() as db_pool: + # Request only 2 records when we have 3 unprocessed records + rows = await db_pool.fetch_and_lock_rows(2) + + # Should respect batch size limit + assert len(rows) == 2 + + # The first 2 records should be returned (ordered by id) + assert rows[0][2] == "issue.updated.0" # event_type + assert rows[1][2] == "issue.updated.1" # event_type + + # Delete the records + delete_outbox_records(processed=False) + + @pytest.mark.asyncio + async def test_fetch_and_lock_rows_empty_when_all_processed( + self, workspace, project + ): + """Test that fetch_and_lock_rows returns empty when all records are processed""" + delete_outbox_records(processed=True) + + # Create processed outbox records + create_outbox_records( + processed_at=timezone.now(), + count=2, + workspace_id=workspace.id, + project_id=project.id, + ) + + async with DatabaseConnectionPool() as db_pool: + # All records in processed_outbox_records fixture are already processed + rows = await db_pool.fetch_and_lock_rows(10) + + # Should return empty list since all records are processed + assert len(rows) == 0 + + # Delete the records + delete_outbox_records(processed=True) + + @pytest.mark.asyncio + async def test_mark_processed_empty_list(self): + """Test mark_processed with empty list of IDs""" + async with DatabaseConnectionPool() as db_pool: + result = await db_pool.mark_processed([]) + + # Should return False for empty list + assert result is False + + @pytest.mark.asyncio + async def test_mark_processed_nonexistent_ids(self): + """Test mark_processed with nonexistent record IDs""" + async with DatabaseConnectionPool() as db_pool: + # Use IDs that don't exist in the database + nonexistent_ids = [99999, 99998] + result = await db_pool.mark_processed(nonexistent_ids) + + # Should return False when no rows are updated + assert result is False + + @pytest.mark.asyncio + async def test_health_check(self): + """Test health_check method returns health status""" + async with DatabaseConnectionPool() as db_pool: + health_status = await db_pool.health_check() + + # Should return a dictionary with health information + assert isinstance(health_status, dict) + assert "healthy" in health_status + assert "timestamp" in health_status + + # Should be healthy for a working connection + assert health_status["healthy"] is True + + @pytest.mark.asyncio + async def test_get_pool_stats(self): + """Test get_pool_stats method returns pool statistics""" + async with DatabaseConnectionPool() as db_pool: + pool_stats = await db_pool.get_pool_stats() + + # Should return a dictionary with pool statistics + assert isinstance(pool_stats, dict) + + # Should contain configuration information + assert "min_size" in pool_stats + assert "max_size" in pool_stats + assert "timeout" in pool_stats + + # Should contain runtime statistics + assert "pool_size" in pool_stats or "error" in pool_stats + + @pytest.mark.asyncio + async def test_health_check_with_no_pool(self): + """Test health_check when pool is not initialized""" + db_pool = DatabaseConnectionPool() + # Don't initialize the pool + health_status = await db_pool.health_check() + + assert health_status["healthy"] is False + assert "Pool not initialized" in health_status["error"] + + @pytest.mark.asyncio + async def test_get_pool_stats_with_no_pool(self): + """Test get_pool_stats when pool is not initialized""" + db_pool = DatabaseConnectionPool() + # Don't initialize the pool + pool_stats = await db_pool.get_pool_stats() + + assert "error" in pool_stats + assert "Pool not initialized" in pool_stats["error"] + + +@pytest.mark.unit +class TestMemoryMonitor: + """Test the MemoryMonitor class""" + + @pytest.mark.asyncio + async def test_memory_monitor_initialization(self): + """Test MemoryMonitor is initialized correctly""" + monitor = MemoryMonitor(memory_limit_mb=100, check_interval=5) + assert monitor.memory_limit_mb == 100 + assert monitor.check_interval == 5 + assert monitor._running is False + + @pytest.mark.asyncio + async def test_memory_monitor_stop(self): + """Test stopping the memory monitor""" + monitor = MemoryMonitor(memory_limit_mb=100, check_interval=5) + monitor._running = True + + await monitor.stop() + assert monitor._running is False + + +@pytest.mark.unit +class TestOutboxModelIntegration: + """Test integration with Outbox model""" + + def test_outbox_model_creation(self, db, workspace, project): + """Test creating outbox records""" + record = Outbox.objects.create( + event_type="issue.created", + entity_type="issue", + entity_id=uuid4(), + payload={"issue_id": "123", "title": "Test Issue"}, + workspace_id=workspace.id, + project_id=project.id, + ) + + assert record.id is not None + assert record.event_id is not None + assert record.event_type == "issue.created" + assert record.entity_type == "issue" + assert record.processed_at is None + assert record.created_at is not None + + def test_outbox_model_processed_records(self, db, workspace, project): + """Test creating processed outbox records""" + record = Outbox.objects.create( + event_type="issue.updated", + entity_type="issue", + entity_id=uuid4(), + payload={"issue_id": "123", "status": "completed"}, + processed_at=timezone.now(), + workspace_id=workspace.id, + project_id=project.id, + ) + + assert record.processed_at is not None + assert record.event_type == "issue.updated" + + def test_outbox_model_string_representation(self, db, workspace, project): + """Test the string representation of Outbox model""" + entity_id = uuid4() + record = Outbox.objects.create( + event_type="issue.created", + entity_type="issue", + entity_id=entity_id, + payload={"issue_id": "123"}, + workspace_id=workspace.id, + project_id=project.id, + ) + + expected_str = f"Outbox {record.id}" + assert str(record) == expected_str diff --git a/apps/api/plane/utils/url.py b/apps/api/plane/utils/url.py index 6c196c2988..5522d46e02 100644 --- a/apps/api/plane/utils/url.py +++ b/apps/api/plane/utils/url.py @@ -35,18 +35,11 @@ def contains_url(value: str) -> bool: bool: True if the string contains a URL, False otherwise """ # Prevent ReDoS by limiting input length - if len(value) > 1000: # Reasonable limit for URL detection + if len(value) > 1000: return False - # Additional safety: truncate very long lines that might contain URLs - lines = value.split("\n") - for line in lines: - if len(line) > 500: # Process only reasonable length lines - line = line[:500] - if URL_PATTERN.search(line): - return True - - return False + # Check for URLs in the entire string + return bool(URL_PATTERN.search(value)) def is_valid_url(url: str) -> bool: diff --git a/apps/api/pytest.ini b/apps/api/pytest.ini index e2f1944567..4a9f2d7ed3 100644 --- a/apps/api/pytest.ini +++ b/apps/api/pytest.ini @@ -9,9 +9,14 @@ markers = contract: Contract tests for API endpoints smoke: Smoke tests for critical functionality slow: Tests that are slow and might be skipped in some contexts + asyncio: Tests that use asyncio addopts = --strict-markers --reuse-db --nomigrations - -vs \ No newline at end of file + -vs + + +# Async test configuration +asyncio_mode = auto \ No newline at end of file diff --git a/apps/api/requirements/base.txt b/apps/api/requirements/base.txt index 62c4a7e17c..c441f8182a 100644 --- a/apps/api/requirements/base.txt +++ b/apps/api/requirements/base.txt @@ -9,6 +9,7 @@ psycopg==3.1.18 psycopg-binary==3.2.3 psycopg-c==3.1.18 dj-database-url==2.1.0 +psycopg-pool==3.2.6 # mongo pymongo==4.6.3 # redis @@ -40,8 +41,6 @@ channels==4.1.0 openai==1.63.2 # slack slack-sdk==3.27.1 -# apm -scout-apm==3.1.0 # xlsx generation openpyxl==3.1.2 # logging @@ -94,3 +93,12 @@ drf-spectacular==0.28.0 # OpenSearch Integration opensearch-py==2.8.0 django-opensearch-dsl==0.7.0 + +# pika for rabbitmq +pika==1.3.2 + +# psutil for monitoring +psutil==7.0.0 + +# pgtriggers +django-pgtrigger==4.15.3 diff --git a/apps/api/requirements/test.txt b/apps/api/requirements/test.txt index 66a1ff1638..c4574a57d0 100644 --- a/apps/api/requirements/test.txt +++ b/apps/api/requirements/test.txt @@ -9,4 +9,5 @@ factory-boy==3.3.0 freezegun==1.2.2 coverage==7.2.7 httpx==0.24.1 -requests==2.32.4 \ No newline at end of file +requests==2.32.4 +pytest-asyncio==1.0.0 \ No newline at end of file