mirror of
https://github.com/makeplane/plane.git
synced 2026-02-25 04:35:21 +01:00
a84d0025a2
* init: automations store and components * chore: updated components * refactor: correct semantics * chore: modal components added * chore: update conjunction translations * Pgtrigger and Outbox prototype * feat: add event stream app with models and migrations - Introduced the event_stream app with initial models: Outbox and IssueProxy. - Implemented a PostgreSQL trigger for logging updates to issues. - Added necessary migrations and updated settings to include the new app. * refactor: migrate event publisher to event stream - Updated imports in test files to reflect the new event_stream app structure. - Removed the event_publisher app, including its models, migrations, and tests. - Introduced MemorySafeOutboxEventListener and related service for handling outbox events. - Added management command for starting the outbox event listener with memory safety. - Implemented PostgreSQL triggers for outbox notifications in the new event_stream app. * refactor: enhance issue serializer and event stream models - Updated IssueSerializer and IssueCreateSerializer to manage assignees and labels more efficiently by calculating additions and removals instead of deleting all and re-adding. - Introduced new proxy models for CycleIssue and ModuleIssue with corresponding PostgreSQL triggers for event handling. - Refactored Outbox model to include entity type and ID for better event tracking. - Removed outdated IssueProxy model and its associated triggers, streamlining the event stream architecture. * chore: update base requirements for OpenSearch and add pika for RabbitMQ - Retained django-opensearch-dsl version 0.7.0 in requirements. - Added pika version 1.3.2 for RabbitMQ integration. * refactor: update import paths for event stream listener - Changed import statements in test_competitive_listeners.py, service.py, and listen_outbox_events.py to reflect the new module structure by removing the 'apiserver' prefix. * feat: implement PostgreSQL LISTEN/NOTIFY listener for outbox events - Added DatabaseConnection class to manage psycopg3 connections for LISTEN/NOTIFY. - Introduced NotificationListener class to handle PostgreSQL NOTIFY messages and dispatch them to registered handlers. - Updated Command class to utilize NotificationListener for listening to outbox events, replacing the previous restartable listener implementation. - Improved error handling and logging throughout the listener process. * feat: enhance event processing with advisory locks - Implemented advisory lock mechanism to prevent concurrent processing of events. - Added methods to acquire and release advisory locks in the DatabaseConnection class. - Updated root_handler to fetch complete outbox data and mark events as processed only if all handlers succeed. - Improved error handling and logging for event processing. * fix: improve event logging and error handling in NotificationListener - Updated event logging to include event ID for better traceability. - Enhanced error handling in the command's handle method to log exceptions and maintain robustness during event listening. * feat: add memory monitoring and auto-restart functionality to event listener - Introduced MemoryMonitor class to track memory usage and event processing counts. - Implemented automatic restarts for the NotificationListener based on memory and event limits. - Enhanced command line options to configure memory limits, event limits, and memory check intervals. - Improved logging for memory stats and event processing to aid in monitoring and debugging. * feat: update local and production settings for event logging - Introduced logging configuration for 'plane.event_stream' in both local and production settings to enhance event logging capabilities. * chore: Implement async outbox polling mechanism with memory monitoring and event handling. Add management command for outbox poller and create necessary database migrations for outbox triggers and indexes. * docs: Add comprehensive README for Event Stream system detailing architecture, components, configuration, usage, and performance tuning. * feat: Enhance MongoDB integration with a singleton connection manager and implement outbox cleanup task for migrating records from PostgreSQL to MongoDB * feat: Add asyncio support for tests and new test files for outbox polling * Merge branch 'event-stream' of github.com:makeplane/plane-ee into event-stream * refactor: Update event types in outbox triggers for cycle, issue, and module models; add IssueComment, IssueLink, and IssueRelation proxies with corresponding triggers * refactor: Update import paths for JsonFormatter and consolidate OpenFeature imports; add unit tests for OutboxPoller and related classes * refactor: Replace all_objects with objects in Outbox queries and update test fixtures for outbox records * feat: Add daily outbox cleaner task to Celery and update type hint for MongoDB collection * refactor: Enhance memory monitoring and outbox polling logic with improved logging and delay handling * feat: Implement outbox triggers for issue, issue attachment, issue link, and issue relation models with dynamic event type determination * chore: Remove obsolete files related to multiple listeners and competitive processing tests * Delete service.py * feat: Add outbox poller script to manage database migrations and process events with configurable parameters * refactor: Optimize EpicCreateSerializer to handle initiative, assignee, and label updates with conflict management and improved logic for adding/removing related entities * feat: Update issue proxy triggers for outbox events - Removed the existing 'issue_outbox_update' trigger from the IssueProxy model. - Added a new 'issue_outbox_update' trigger with enhanced logic to handle conversion events when the issue type changes. - The new trigger captures updates and soft deletes, inserting appropriate events into the outbox. * feat: Refactor issue proxy triggers and add new outbox event handling - Removed and replaced existing triggers for IssueProxy, IssueAssigneeProxy, IssueLabelProxy, CycleIssueProxy, and ModuleIssueProxy. - Introduced new triggers to handle outbox events for various issue-related actions, including creation, updates, and deletions. - Enhanced logic for event type determination based on issue type, ensuring accurate event handling in the outbox. * refactor: Update serializers to use ID fields for assignees and labels - Changed the handling of assignees and labels in IssueCreateSerializer and EpicCreateSerializer to use ID fields instead of object references. - Improved logic for determining assignees and labels to add or remove by using lists of IDs. - Enhanced code readability with comments explaining the changes. * feat: Enhance issue event handling in outbox triggers - Updated triggers for IssueProxy, IssueAssigneeProxy, and IssueLabelProxy to determine event types based on issue type, including handling for epics. - Introduced filtered data for outbox events to exclude description fields, improving data integrity and reducing unnecessary payload size. - Enhanced logic for detecting changes during updates, ensuring only relevant changes are captured and processed. * feat: implement connection pooling for outbox poller - Introduced DatabaseConnectionPool class to manage async connection pooling using psycopg_pool. - Enhanced outbox polling logic to utilize connection pooling for improved performance and resource management. - Added health check and statistics retrieval methods for the connection pool. - Updated outbox model and migration to include claimed_at field for better event tracking. - Refactored tests to validate new connection pooling functionality. * feat: Introduce automation engine with models and registry - Added core components for the automation engine, including models for Automation, AutomationVersion, AutomationNode, AutomationEdge, AutomationRun, and NodeExecution. - Implemented a NodeRegistry for managing automation nodes and a decorator for easy registration. - Created initial files for automation app configuration and documentation, including an entity-relationship diagram and detailed model explanations. - Established a migration for the new automation models to integrate with the existing database schema. * refactor: move MongoConnection to a new settings module and update imports - Created a new mongo.py file to define the MongoConnection class for managing MongoDB connections. - Updated the import path in outbox_cleaner.py to reference the new MongoConnection location. * refactor: update import path for MongoConnection in outbox_cleaner.py * chore: remove scout_apm from production settings and requirements - Deleted scout_apm from the installed apps in production.py. - Removed scout-apm dependency from base.txt requirements. * feat: add claimed_at field to outbox events and update requirements - Added claimed_at field to the outbox event model for enhanced event tracking. - Updated outbox poller to handle claimed_at in event processing. - Included django-pgtrigger in base.txt requirements for database trigger management. * feat: enhance memory monitoring in outbox poller - Updated MemoryMonitor to signal for restarts instead of exiting on memory limit exceedance. - Added methods to check for restart requests and wait for restart signals. - Improved outbox poller logic to handle memory monitoring more gracefully during processing. - Ensured all claimed rows are processed before initiating a restart. * feat: implement graceful shutdown handling in outbox poller - Added GracefulShutdownHandler class to manage shutdown signals (SIGTERM, SIGINT, SIGQUIT). - Integrated shutdown handling into the outbox poller to allow for graceful exits during processing. - Updated polling logic to check for shutdown requests and clean up resources accordingly. - Enhanced command help text to reflect new signal handling capabilities. * simplify AutomationEdge model by removing label and condition fields, and update unique constraint * Enhance outbox event handling by adding workspace and project IDs - Updated the outbox cleaner to include `workspace_id` and `project_id` in the deletion process. - Modified the outbox poller to handle new fields in event processing. - Adjusted models and migrations to support the new fields in the outbox table. - Updated tests to ensure proper handling of workspace and project IDs in outbox records. * chore: folder migration * fix: type errors * Refactor AutomationNode model and migration - Removed `node_id`, `position_x`, `position_y`, and `execution_order` fields from the AutomationNode model. - Updated migration to reflect the removal of constraints and adjusted the ordering of the model. - Cleaned up the migration dependencies for better clarity. * Add automation serializers and endpoints - Introduced AutomationWriteSerializer and AutomationReadSerializer for handling automation data. - Added AutomationNodeWriteSerializer and AutomationNodeReadSerializer for automation node management. - Created AutomationEndpoint and AutomationNodeEndpoint views to manage automation and automation nodes with CRUD operations. - Implemented permission checks for different roles on the endpoints. * Merge branch 'preview' of github.com:makeplane/plane-ee into event-stream * Add automation scope choices and update serializers and views - Introduced AutomationScopeChoices for defining automation scopes. - Updated AutomationWriteSerializer and AutomationReadSerializer to include scope validation and new fields. - Created automation URL patterns and endpoints for managing automations. - Implemented feature flag checks for automation-related endpoints. - Enhanced permission checks for CRUD operations on automations. * Refactor automation models, serializers, and views for improved structure and functionality - Updated migration file to enhance automation-related models, including Automation, AutomationVersion, AutomationRun, AutomationNode, and AutomationEdge. - Refactored serializers to include additional fields and validation for AutomationNode. - Introduced new endpoints for managing AutomationNodes, including CRUD operations with feature flag checks. - Improved URL patterns for automation and automation node management. - Enhanced permission checks for automation-related operations. * Add AutomationEdge serializers and endpoints for managing automation edges - Introduced AutomationEdgeWriteSerializer and AutomationEdgeReadSerializer for handling automation edge data. - Created AutomationEdgeEndpoint to manage CRUD operations for automation edges with appropriate feature flag checks. - Updated URL patterns to include routes for automation edges. - Refactored existing automation node endpoint to inherit from AutomationBaseEndpoint for consistency. * feat: add Automation Bot type and enhance automation model - Introduced a new BotTypeEnum value for "AUTOMATION_BOT". - Added a OneToOneField for bot_user in the Automation model to associate a bot user with automations. - Created a new AutomationActivity model to track activities related to automations, including fields for action, old and new values, and related entities. * feat: add automation models and constraints - Introduced new models for Automation, AutomationNode, AutomationRun, NodeExecution, AutomationVersion, AutomationEdge, and AutomationActivity. - Implemented foreign key relationships and constraints to ensure data integrity, including unique constraints with conditions for deleted records. - Added indexes to optimize query performance on various fields across the new models. - Enhanced existing models with additional fields and relationships to support automation functionality. * feat: rich filters store architecture and UI components * improvement: use map instead of object for automations store * improvement: rename automations v2 components to automations * feat: automation nodes and edges * feat: enhance automation functionality with new models and endpoints - Added AutomationRun and NodeExecution models to track automation runs and their execution details. - Introduced AutomationDetailReadSerializer to provide detailed information about automations, including nodes and edges. - Updated AutomationEndpoint to create and manage automation bots, enhancing automation capabilities. - Created AutomationRunEndpoint for retrieving automation run data, supporting both single and multiple run queries. - Refactored existing serializers and views to incorporate new features and improve overall structure. * improvement: refactor automation nodes * feat: publisher for pushing events from the poller to the queue * feat: Implement RabbitMQ Event Stream Publisher and Outbox Event Model - Added EventStreamPublisher class for reliable event publishing to RabbitMQ with support for fanout exchanges, message persistence, and concurrent access handling. - Introduced OutboxEvent dataclass for structured representation of outbox events, including methods for conversion to various formats. - Updated OutboxPoller to utilize the new publisher for handling outbox events, improving event processing and publishing logic. * feat: Add logging for published messages in EventStreamPublisher - Enhanced the EventStreamPublisher class to log published messages, including instance ID and message body for better traceability and debugging. * feat: Enhance Outbox Poller Tests and Introduce Event Stream Publisher Tests - Refactored test cases in `test_outbox_poller.py` to simplify row tuple creation and improve readability. - Added new test suite for `EventStreamPublisher` in `test_publisher.py`, covering initialization, connection management, message publishing, error handling, and thread safety. - Introduced a new `__init__.py` file in the event stream tests directory to support module structure. --------- Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com> * feat: implement automation activity tracking functionality - Added a new module for tracking automation activities, including creation, updates, and deletions. - Introduced functions to handle different types of automation activities and log changes to relevant fields. - Updated AutomationEndpoint to trigger automation activity tasks upon creation, update, and deletion of automations. - Enhanced error handling and logging for automation activity processes. * feat: expand automation activity tracking with node and edge functionalities - Added functions to track creation, updates, and deletions of automation nodes and edges. - Implemented new activity types for node and edge activities in the automation activity task. - Updated AutomationNodeEndpoint and AutomationEdgeEndpoint to trigger activity tracking upon relevant actions. - Enhanced serialization of requested data for activity logging, ensuring accurate tracking of changes. * feat: add automation activity tracking for activities - Introduced AutomationActivityReadSerializer to handle serialization of automation activity data. - Created AutomationActivityEndpoint for managing CRUD operations related to automation activities. - Updated URL patterns to include routes for automation activities. - Enhanced existing serializers and views to support new activity tracking functionalities. * feat: add unit tests for automation activity tracking - Introduced comprehensive unit tests for automation activity functions, including creation, deletion, and updates of automation activities, nodes, and edges. - Implemented test cases to validate the tracking of field changes and ensure accurate logging of activities. - Enhanced test coverage for various scenarios, including valid and invalid data inputs, to ensure robustness of automation activity tracking functionalities. * feat: Implement automation execution engine and node management - Added AutomationExecutionEngine class to handle the execution of automation workflows, including validation of events, loading of nodes, and execution of triggers, conditions, and actions. - Introduced various node types (triggers, conditions, actions) with specific implementations for handling events and performing actions within the automation framework. - Enhanced the automation system with auto-discovery of node modules during Django startup, ensuring all nodes are registered and ready for execution. - Created comprehensive README documentation for automation nodes, detailing available triggers, conditions, and their configurations. - Added error handling and logging for improved traceability during automation execution. - Implemented tests for the new automation features to ensure reliability and correctness. This commit lays the foundation for a robust automation system capable of responding to database events and executing defined workflows. * Added automations to installed apps * refactor: remove unused import in automation activity test file * chore: automations components restructure and API integration init * improvement: add loader for automations list table * fix: automations settings page layout * fix: automation list loader * improvement: reduce add node button height * feat: update automation modal. * refactor: simplify label ID handling in IssueCreateSerializer * improvement: rename `actions` to `action-buttons` in automation details sidebar. * feat: add automation status endpoint to toggle status for different automations * feat: automation comment actions * chore: default last run status to None * chore: add total success count * chore: add total failed count * improvement: automations list table * chore: automation feature flag update and minor fix in status endpoint * fix: automation status endpoint feature flag * feat: automation details page header * chore: remove support for multiple trigger nodes in a single automation * feat: update automation status handling to include is_enabled flag * improvements: minor enhancements for rich filters * added button configuration to add filters button * added option to disable transition for filter item * fixed filter item label truncation * added support for changing work item filter config keys * improvements: add filter button config initialization * chore: minor type name updates * Updated migration * Implement automation consumer and management commands for event processing - Added AutomationConsumer class to handle RabbitMQ events and dispatch to Celery tasks. - Created management commands for running the automation consumer and creating automations interactively. - Introduced Celery tasks for executing automation logic and cleaning up processed events. - Updated settings to include automation-related configurations. - Implemented logging for better traceability of automation processes. * Enhance automation execution engine with structured results and multiple action support - Introduced new data classes for standardized execution context and results: ExecutionContext, NodeResult, and AutomationResult. - Updated AutomationExecutionEngine to support multiple action nodes executed sequentially, halting on the first failure. - Refactored node execution methods to improve clarity and maintainability. - Enhanced error handling and logging for better traceability during automation execution. - Updated README to reflect changes in automation execution flow and node capabilities. * feat: enhance automation model to create bot user and update field tracking * feat: add initiator_id field to Outbox model and update migration dependencies * feat: condition node for automations * chore: enable feature flag check for automation create and update status endpoint * feat: add initiator_id to Outbox insertions across event stream models - Updated Outbox insert statements in various models to include initiator_id. - Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure proper tracking of the user initiating changes. - Enhanced event handling for soft deletes and regular updates to maintain consistency in data tracking. * feat: enable / disable automations * feat: include initiator_id in OutboxEvent model and database interactions - Added initiator_id field to the OutboxEvent model for enhanced tracking. - Updated database queries in outbox_poller.py to include initiator_id in insertions. - Ensured consistency in data handling across event stream components. * refactor: update handle_row function to use OutboxEvent model - Changed the parameter type of handle_row from Dict to OutboxEvent for better type safety. - Updated logging to utilize the to_dict method of OutboxEvent for consistent event data representation. * chore: update `edgeIdsToDelete` to `edgeIdsToRemove` * refactor: simplify EventStreamPublisher by removing default queue and message TTL parameters, and updating setup methods for exchange only * feat: add change property action * TODOs * chore: automation activity * chore: added issue for issue created trigger * chore: updated migrations * chore: added priority, date and state changes * chore: update automation migration * chore: remove label activity * chore: created the comment activity * fix: logical operators casing * feat: add docker entrypoint script for automation consumer - Introduced a new script to manage the automation consumer process. - The script includes commands to wait for the database and migrations before running the automation consumer. * feat: add automation models and migrations - Introduced new models for Automation, AutomationActivity, AutomationEdge, AutomationNode, AutomationRun, AutomationVersion, NodeExecution, and ProcessedAutomationEvent. - Implemented necessary migrations to support the new automation features, including relationships and constraints. - Enhanced event handling capabilities with detailed tracking and management of automation processes. * chore: trigger main content condition preview * chore: update condition preview limit * chore: validated conditions for automations * chore: added check for automation and import job * refactor: streamline event processing by ignoring system-originated events * fix: automation activities * refactor: update SQL command for setting initiator type in automation and import tasks * chore: update lxml and xmlsec versions in base requirements * chore: update outbox cleaner cutoff days to 7 and include it in settings * fix: update SQL command for setting initiator type in data import and bulk update tasks * feat: add triggers for issue assignee and label management in outbox - Implemented triggers for inserting and updating issue assignee and label data in the outbox. - Adjusted SQL logic to handle previous attributes correctly during assignee and label updates. - Changed trigger execution timing from AFTER to BEFORE for better handling of soft deletes. * feat: enhance automation consumer script with queue and prefetch options - Added support for specifying the automation events queue and prefetch count in the docker entrypoint script for the automation consumer. * fix: update automation consumer script and event stream exchange name - Changed the queue name in the automation consumer script to use the new environment variable for event stream. - Updated the default exchange name in the event stream publisher to align with the new naming convention. * refactor: update automation consumer settings and permissions - Changed the queue name and exchange name in the automation consumer and related scripts to use new environment variables. - Updated file permissions for the docker entrypoint script to ensure proper execution. * chore: activity filters * fix: activity filters * chore: automation activity description * chore: added new fields in automation run * improvement: add custom automations to project settings page * improvement: minor enhancements and fixes. * feat: update automation consumer settings and logging configuration * Changed default queue name for automation events in run_automation_consumer.py. * Added logging configuration for plane.automations.consumer in both local and production settings. * Removed unused node_execution.py file. * minor cleanup * chore: minor improvements * feat: update automation execution tracking and logging * Added logic to update the last run timestamp and increment the run count for automations. * Enhanced the field tracking in automation activity logging to include node type for better clarity. --------- Co-authored-by: Aaryan Khandelwal <aaryankhandu123@gmail.com> Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com> Co-authored-by: pablohashescobar <nikhilschacko@gmail.com> Co-authored-by: Nikhil <118773738+pablohashescobar@users.noreply.github.com> Co-authored-by: NarayanBavisetti <narayan3119@gmail.com> Co-authored-by: Bavisetti Narayan <72156168+NarayanBavisetti@users.noreply.github.com>