mirror of
https://github.com/makeplane/plane.git
synced 2026-07-13 14:01:45 +02:00
a84d0025a2
* init: automations store and components * chore: updated components * refactor: correct semantics * chore: modal components added * chore: update conjunction translations * Pgtrigger and Outbox prototype * feat: add event stream app with models and migrations - Introduced the event_stream app with initial models: Outbox and IssueProxy. - Implemented a PostgreSQL trigger for logging updates to issues. - Added necessary migrations and updated settings to include the new app. * refactor: migrate event publisher to event stream - Updated imports in test files to reflect the new event_stream app structure. - Removed the event_publisher app, including its models, migrations, and tests. - Introduced MemorySafeOutboxEventListener and related service for handling outbox events. - Added management command for starting the outbox event listener with memory safety. - Implemented PostgreSQL triggers for outbox notifications in the new event_stream app. * refactor: enhance issue serializer and event stream models - Updated IssueSerializer and IssueCreateSerializer to manage assignees and labels more efficiently by calculating additions and removals instead of deleting all and re-adding. - Introduced new proxy models for CycleIssue and ModuleIssue with corresponding PostgreSQL triggers for event handling. - Refactored Outbox model to include entity type and ID for better event tracking. - Removed outdated IssueProxy model and its associated triggers, streamlining the event stream architecture. * chore: update base requirements for OpenSearch and add pika for RabbitMQ - Retained django-opensearch-dsl version 0.7.0 in requirements. - Added pika version 1.3.2 for RabbitMQ integration. * refactor: update import paths for event stream listener - Changed import statements in test_competitive_listeners.py, service.py, and listen_outbox_events.py to reflect the new module structure by removing the 'apiserver' prefix. * feat: implement PostgreSQL LISTEN/NOTIFY listener for outbox events - Added DatabaseConnection class to manage psycopg3 connections for LISTEN/NOTIFY. - Introduced NotificationListener class to handle PostgreSQL NOTIFY messages and dispatch them to registered handlers. - Updated Command class to utilize NotificationListener for listening to outbox events, replacing the previous restartable listener implementation. - Improved error handling and logging throughout the listener process. * feat: enhance event processing with advisory locks - Implemented advisory lock mechanism to prevent concurrent processing of events. - Added methods to acquire and release advisory locks in the DatabaseConnection class. - Updated root_handler to fetch complete outbox data and mark events as processed only if all handlers succeed. - Improved error handling and logging for event processing. * fix: improve event logging and error handling in NotificationListener - Updated event logging to include event ID for better traceability. - Enhanced error handling in the command's handle method to log exceptions and maintain robustness during event listening. * feat: add memory monitoring and auto-restart functionality to event listener - Introduced MemoryMonitor class to track memory usage and event processing counts. - Implemented automatic restarts for the NotificationListener based on memory and event limits. - Enhanced command line options to configure memory limits, event limits, and memory check intervals. - Improved logging for memory stats and event processing to aid in monitoring and debugging. * feat: update local and production settings for event logging - Introduced logging configuration for 'plane.event_stream' in both local and production settings to enhance event logging capabilities. * chore: Implement async outbox polling mechanism with memory monitoring and event handling. Add management command for outbox poller and create necessary database migrations for outbox triggers and indexes. * docs: Add comprehensive README for Event Stream system detailing architecture, components, configuration, usage, and performance tuning. * feat: Enhance MongoDB integration with a singleton connection manager and implement outbox cleanup task for migrating records from PostgreSQL to MongoDB * feat: Add asyncio support for tests and new test files for outbox polling * Merge branch 'event-stream' of github.com:makeplane/plane-ee into event-stream * refactor: Update event types in outbox triggers for cycle, issue, and module models; add IssueComment, IssueLink, and IssueRelation proxies with corresponding triggers * refactor: Update import paths for JsonFormatter and consolidate OpenFeature imports; add unit tests for OutboxPoller and related classes * refactor: Replace all_objects with objects in Outbox queries and update test fixtures for outbox records * feat: Add daily outbox cleaner task to Celery and update type hint for MongoDB collection * refactor: Enhance memory monitoring and outbox polling logic with improved logging and delay handling * feat: Implement outbox triggers for issue, issue attachment, issue link, and issue relation models with dynamic event type determination * chore: Remove obsolete files related to multiple listeners and competitive processing tests * Delete service.py * feat: Add outbox poller script to manage database migrations and process events with configurable parameters * refactor: Optimize EpicCreateSerializer to handle initiative, assignee, and label updates with conflict management and improved logic for adding/removing related entities * feat: Update issue proxy triggers for outbox events - Removed the existing 'issue_outbox_update' trigger from the IssueProxy model. - Added a new 'issue_outbox_update' trigger with enhanced logic to handle conversion events when the issue type changes. - The new trigger captures updates and soft deletes, inserting appropriate events into the outbox. * feat: Refactor issue proxy triggers and add new outbox event handling - Removed and replaced existing triggers for IssueProxy, IssueAssigneeProxy, IssueLabelProxy, CycleIssueProxy, and ModuleIssueProxy. - Introduced new triggers to handle outbox events for various issue-related actions, including creation, updates, and deletions. - Enhanced logic for event type determination based on issue type, ensuring accurate event handling in the outbox. * refactor: Update serializers to use ID fields for assignees and labels - Changed the handling of assignees and labels in IssueCreateSerializer and EpicCreateSerializer to use ID fields instead of object references. - Improved logic for determining assignees and labels to add or remove by using lists of IDs. - Enhanced code readability with comments explaining the changes. * feat: Enhance issue event handling in outbox triggers - Updated triggers for IssueProxy, IssueAssigneeProxy, and IssueLabelProxy to determine event types based on issue type, including handling for epics. - Introduced filtered data for outbox events to exclude description fields, improving data integrity and reducing unnecessary payload size. - Enhanced logic for detecting changes during updates, ensuring only relevant changes are captured and processed. * feat: implement connection pooling for outbox poller - Introduced DatabaseConnectionPool class to manage async connection pooling using psycopg_pool. - Enhanced outbox polling logic to utilize connection pooling for improved performance and resource management. - Added health check and statistics retrieval methods for the connection pool. - Updated outbox model and migration to include claimed_at field for better event tracking. - Refactored tests to validate new connection pooling functionality. * feat: Introduce automation engine with models and registry - Added core components for the automation engine, including models for Automation, AutomationVersion, AutomationNode, AutomationEdge, AutomationRun, and NodeExecution. - Implemented a NodeRegistry for managing automation nodes and a decorator for easy registration. - Created initial files for automation app configuration and documentation, including an entity-relationship diagram and detailed model explanations. - Established a migration for the new automation models to integrate with the existing database schema. * refactor: move MongoConnection to a new settings module and update imports - Created a new mongo.py file to define the MongoConnection class for managing MongoDB connections. - Updated the import path in outbox_cleaner.py to reference the new MongoConnection location. * refactor: update import path for MongoConnection in outbox_cleaner.py * chore: remove scout_apm from production settings and requirements - Deleted scout_apm from the installed apps in production.py. - Removed scout-apm dependency from base.txt requirements. * feat: add claimed_at field to outbox events and update requirements - Added claimed_at field to the outbox event model for enhanced event tracking. - Updated outbox poller to handle claimed_at in event processing. - Included django-pgtrigger in base.txt requirements for database trigger management. * feat: enhance memory monitoring in outbox poller - Updated MemoryMonitor to signal for restarts instead of exiting on memory limit exceedance. - Added methods to check for restart requests and wait for restart signals. - Improved outbox poller logic to handle memory monitoring more gracefully during processing. - Ensured all claimed rows are processed before initiating a restart. * feat: implement graceful shutdown handling in outbox poller - Added GracefulShutdownHandler class to manage shutdown signals (SIGTERM, SIGINT, SIGQUIT). - Integrated shutdown handling into the outbox poller to allow for graceful exits during processing. - Updated polling logic to check for shutdown requests and clean up resources accordingly. - Enhanced command help text to reflect new signal handling capabilities. * simplify AutomationEdge model by removing label and condition fields, and update unique constraint * Enhance outbox event handling by adding workspace and project IDs - Updated the outbox cleaner to include `workspace_id` and `project_id` in the deletion process. - Modified the outbox poller to handle new fields in event processing. - Adjusted models and migrations to support the new fields in the outbox table. - Updated tests to ensure proper handling of workspace and project IDs in outbox records. * chore: folder migration * fix: type errors * Refactor AutomationNode model and migration - Removed `node_id`, `position_x`, `position_y`, and `execution_order` fields from the AutomationNode model. - Updated migration to reflect the removal of constraints and adjusted the ordering of the model. - Cleaned up the migration dependencies for better clarity. * Add automation serializers and endpoints - Introduced AutomationWriteSerializer and AutomationReadSerializer for handling automation data. - Added AutomationNodeWriteSerializer and AutomationNodeReadSerializer for automation node management. - Created AutomationEndpoint and AutomationNodeEndpoint views to manage automation and automation nodes with CRUD operations. - Implemented permission checks for different roles on the endpoints. * Merge branch 'preview' of github.com:makeplane/plane-ee into event-stream * Add automation scope choices and update serializers and views - Introduced AutomationScopeChoices for defining automation scopes. - Updated AutomationWriteSerializer and AutomationReadSerializer to include scope validation and new fields. - Created automation URL patterns and endpoints for managing automations. - Implemented feature flag checks for automation-related endpoints. - Enhanced permission checks for CRUD operations on automations. * Refactor automation models, serializers, and views for improved structure and functionality - Updated migration file to enhance automation-related models, including Automation, AutomationVersion, AutomationRun, AutomationNode, and AutomationEdge. - Refactored serializers to include additional fields and validation for AutomationNode. - Introduced new endpoints for managing AutomationNodes, including CRUD operations with feature flag checks. - Improved URL patterns for automation and automation node management. - Enhanced permission checks for automation-related operations. * Add AutomationEdge serializers and endpoints for managing automation edges - Introduced AutomationEdgeWriteSerializer and AutomationEdgeReadSerializer for handling automation edge data. - Created AutomationEdgeEndpoint to manage CRUD operations for automation edges with appropriate feature flag checks. - Updated URL patterns to include routes for automation edges. - Refactored existing automation node endpoint to inherit from AutomationBaseEndpoint for consistency. * feat: add Automation Bot type and enhance automation model - Introduced a new BotTypeEnum value for "AUTOMATION_BOT". - Added a OneToOneField for bot_user in the Automation model to associate a bot user with automations. - Created a new AutomationActivity model to track activities related to automations, including fields for action, old and new values, and related entities. * feat: add automation models and constraints - Introduced new models for Automation, AutomationNode, AutomationRun, NodeExecution, AutomationVersion, AutomationEdge, and AutomationActivity. - Implemented foreign key relationships and constraints to ensure data integrity, including unique constraints with conditions for deleted records. - Added indexes to optimize query performance on various fields across the new models. - Enhanced existing models with additional fields and relationships to support automation functionality. * feat: rich filters store architecture and UI components * improvement: use map instead of object for automations store * improvement: rename automations v2 components to automations * feat: automation nodes and edges * feat: enhance automation functionality with new models and endpoints - Added AutomationRun and NodeExecution models to track automation runs and their execution details. - Introduced AutomationDetailReadSerializer to provide detailed information about automations, including nodes and edges. - Updated AutomationEndpoint to create and manage automation bots, enhancing automation capabilities. - Created AutomationRunEndpoint for retrieving automation run data, supporting both single and multiple run queries. - Refactored existing serializers and views to incorporate new features and improve overall structure. * improvement: refactor automation nodes * feat: publisher for pushing events from the poller to the queue * feat: Implement RabbitMQ Event Stream Publisher and Outbox Event Model - Added EventStreamPublisher class for reliable event publishing to RabbitMQ with support for fanout exchanges, message persistence, and concurrent access handling. - Introduced OutboxEvent dataclass for structured representation of outbox events, including methods for conversion to various formats. - Updated OutboxPoller to utilize the new publisher for handling outbox events, improving event processing and publishing logic. * feat: Add logging for published messages in EventStreamPublisher - Enhanced the EventStreamPublisher class to log published messages, including instance ID and message body for better traceability and debugging. * feat: Enhance Outbox Poller Tests and Introduce Event Stream Publisher Tests - Refactored test cases in `test_outbox_poller.py` to simplify row tuple creation and improve readability. - Added new test suite for `EventStreamPublisher` in `test_publisher.py`, covering initialization, connection management, message publishing, error handling, and thread safety. - Introduced a new `__init__.py` file in the event stream tests directory to support module structure. --------- Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com> * feat: implement automation activity tracking functionality - Added a new module for tracking automation activities, including creation, updates, and deletions. - Introduced functions to handle different types of automation activities and log changes to relevant fields. - Updated AutomationEndpoint to trigger automation activity tasks upon creation, update, and deletion of automations. - Enhanced error handling and logging for automation activity processes. * feat: expand automation activity tracking with node and edge functionalities - Added functions to track creation, updates, and deletions of automation nodes and edges. - Implemented new activity types for node and edge activities in the automation activity task. - Updated AutomationNodeEndpoint and AutomationEdgeEndpoint to trigger activity tracking upon relevant actions. - Enhanced serialization of requested data for activity logging, ensuring accurate tracking of changes. * feat: add automation activity tracking for activities - Introduced AutomationActivityReadSerializer to handle serialization of automation activity data. - Created AutomationActivityEndpoint for managing CRUD operations related to automation activities. - Updated URL patterns to include routes for automation activities. - Enhanced existing serializers and views to support new activity tracking functionalities. * feat: add unit tests for automation activity tracking - Introduced comprehensive unit tests for automation activity functions, including creation, deletion, and updates of automation activities, nodes, and edges. - Implemented test cases to validate the tracking of field changes and ensure accurate logging of activities. - Enhanced test coverage for various scenarios, including valid and invalid data inputs, to ensure robustness of automation activity tracking functionalities. * feat: Implement automation execution engine and node management - Added AutomationExecutionEngine class to handle the execution of automation workflows, including validation of events, loading of nodes, and execution of triggers, conditions, and actions. - Introduced various node types (triggers, conditions, actions) with specific implementations for handling events and performing actions within the automation framework. - Enhanced the automation system with auto-discovery of node modules during Django startup, ensuring all nodes are registered and ready for execution. - Created comprehensive README documentation for automation nodes, detailing available triggers, conditions, and their configurations. - Added error handling and logging for improved traceability during automation execution. - Implemented tests for the new automation features to ensure reliability and correctness. This commit lays the foundation for a robust automation system capable of responding to database events and executing defined workflows. * Added automations to installed apps * refactor: remove unused import in automation activity test file * chore: automations components restructure and API integration init * improvement: add loader for automations list table * fix: automations settings page layout * fix: automation list loader * improvement: reduce add node button height * feat: update automation modal. * refactor: simplify label ID handling in IssueCreateSerializer * improvement: rename `actions` to `action-buttons` in automation details sidebar. * feat: add automation status endpoint to toggle status for different automations * feat: automation comment actions * chore: default last run status to None * chore: add total success count * chore: add total failed count * improvement: automations list table * chore: automation feature flag update and minor fix in status endpoint * fix: automation status endpoint feature flag * feat: automation details page header * chore: remove support for multiple trigger nodes in a single automation * feat: update automation status handling to include is_enabled flag * improvements: minor enhancements for rich filters * added button configuration to add filters button * added option to disable transition for filter item * fixed filter item label truncation * added support for changing work item filter config keys * improvements: add filter button config initialization * chore: minor type name updates * Updated migration * Implement automation consumer and management commands for event processing - Added AutomationConsumer class to handle RabbitMQ events and dispatch to Celery tasks. - Created management commands for running the automation consumer and creating automations interactively. - Introduced Celery tasks for executing automation logic and cleaning up processed events. - Updated settings to include automation-related configurations. - Implemented logging for better traceability of automation processes. * Enhance automation execution engine with structured results and multiple action support - Introduced new data classes for standardized execution context and results: ExecutionContext, NodeResult, and AutomationResult. - Updated AutomationExecutionEngine to support multiple action nodes executed sequentially, halting on the first failure. - Refactored node execution methods to improve clarity and maintainability. - Enhanced error handling and logging for better traceability during automation execution. - Updated README to reflect changes in automation execution flow and node capabilities. * feat: enhance automation model to create bot user and update field tracking * feat: add initiator_id field to Outbox model and update migration dependencies * feat: condition node for automations * chore: enable feature flag check for automation create and update status endpoint * feat: add initiator_id to Outbox insertions across event stream models - Updated Outbox insert statements in various models to include initiator_id. - Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure proper tracking of the user initiating changes. - Enhanced event handling for soft deletes and regular updates to maintain consistency in data tracking. * feat: enable / disable automations * feat: include initiator_id in OutboxEvent model and database interactions - Added initiator_id field to the OutboxEvent model for enhanced tracking. - Updated database queries in outbox_poller.py to include initiator_id in insertions. - Ensured consistency in data handling across event stream components. * refactor: update handle_row function to use OutboxEvent model - Changed the parameter type of handle_row from Dict to OutboxEvent for better type safety. - Updated logging to utilize the to_dict method of OutboxEvent for consistent event data representation. * chore: update `edgeIdsToDelete` to `edgeIdsToRemove` * refactor: simplify EventStreamPublisher by removing default queue and message TTL parameters, and updating setup methods for exchange only * feat: add change property action * TODOs * chore: automation activity * chore: added issue for issue created trigger * chore: updated migrations * chore: added priority, date and state changes * chore: update automation migration * chore: remove label activity * chore: created the comment activity * fix: logical operators casing * feat: add docker entrypoint script for automation consumer - Introduced a new script to manage the automation consumer process. - The script includes commands to wait for the database and migrations before running the automation consumer. * feat: add automation models and migrations - Introduced new models for Automation, AutomationActivity, AutomationEdge, AutomationNode, AutomationRun, AutomationVersion, NodeExecution, and ProcessedAutomationEvent. - Implemented necessary migrations to support the new automation features, including relationships and constraints. - Enhanced event handling capabilities with detailed tracking and management of automation processes. * chore: trigger main content condition preview * chore: update condition preview limit * chore: validated conditions for automations * chore: added check for automation and import job * refactor: streamline event processing by ignoring system-originated events * fix: automation activities * refactor: update SQL command for setting initiator type in automation and import tasks * chore: update lxml and xmlsec versions in base requirements * chore: update outbox cleaner cutoff days to 7 and include it in settings * fix: update SQL command for setting initiator type in data import and bulk update tasks * feat: add triggers for issue assignee and label management in outbox - Implemented triggers for inserting and updating issue assignee and label data in the outbox. - Adjusted SQL logic to handle previous attributes correctly during assignee and label updates. - Changed trigger execution timing from AFTER to BEFORE for better handling of soft deletes. * feat: enhance automation consumer script with queue and prefetch options - Added support for specifying the automation events queue and prefetch count in the docker entrypoint script for the automation consumer. * fix: update automation consumer script and event stream exchange name - Changed the queue name in the automation consumer script to use the new environment variable for event stream. - Updated the default exchange name in the event stream publisher to align with the new naming convention. * refactor: update automation consumer settings and permissions - Changed the queue name and exchange name in the automation consumer and related scripts to use new environment variables. - Updated file permissions for the docker entrypoint script to ensure proper execution. * chore: activity filters * fix: activity filters * chore: automation activity description * chore: added new fields in automation run * improvement: add custom automations to project settings page * improvement: minor enhancements and fixes. * feat: update automation consumer settings and logging configuration * Changed default queue name for automation events in run_automation_consumer.py. * Added logging configuration for plane.automations.consumer in both local and production settings. * Removed unused node_execution.py file. * minor cleanup * chore: minor improvements * feat: update automation execution tracking and logging * Added logic to update the last run timestamp and increment the run count for automations. * Enhanced the field tracking in automation activity logging to include node type for better clarity. --------- Co-authored-by: Aaryan Khandelwal <aaryankhandu123@gmail.com> Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com> Co-authored-by: pablohashescobar <nikhilschacko@gmail.com> Co-authored-by: Nikhil <118773738+pablohashescobar@users.noreply.github.com> Co-authored-by: NarayanBavisetti <narayan3119@gmail.com> Co-authored-by: Bavisetti Narayan <72156168+NarayanBavisetti@users.noreply.github.com>
Modern project management for all teams
Website • Releases • Twitter • Documentation
Meet Plane, an open-source project management tool to track issues, run sprints cycles, and manage product roadmaps without the chaos of managing the tool itself. 🧘♀️
Plane is evolving every day. Your suggestions, ideas, and reported bugs help us immensely. Do not hesitate to join in the conversation on Discord or raise a GitHub issue. We read everything and respond to most.
🚀 Installation
Getting started with Plane is simple. Choose the setup that works best for you:
-
Plane Cloud Sign up for a free account on Plane Cloud—it's the fastest way to get up and running without worrying about infrastructure.
-
Self-host Plane Prefer full control over your data and infrastructure? Install and run Plane on your own servers. Follow our detailed deployment guides to get started.
| Installation methods | Docs link |
|---|---|
| Docker |  |
| Kubernetes |  |
Instance admins can configure instance settings with God mode.
🌟 Features
-
Issues Efficiently create and manage tasks with a robust rich text editor that supports file uploads. Enhance organization and tracking by adding sub-properties and referencing related issues.
-
Cycles Maintain your team’s momentum with Cycles. Track progress effortlessly using burn-down charts and other insightful tools.
-
Modules Simplify complex projects by dividing them into smaller, manageable modules.
-
Views Customize your workflow by creating filters to display only the most relevant issues. Save and share these views with ease.
-
Pages Capture and organize ideas using Plane Pages, complete with AI capabilities and a rich text editor. Format text, insert images, add hyperlinks, or convert your notes into actionable items.
-
Analytics Access real-time insights across all your Plane data. Visualize trends, remove blockers, and keep your projects moving forward.
-
Drive (coming soon): The drive helps you share documents, images, videos, or any other files that make sense to you or your team and align on the problem/solution.
🛠️ Local development
See CONTRIBUTING
⚙️ Built with
📸 Screenshots
📝 Documentation
Explore Plane's product documentation and developer documentation to learn about features, setup, and usage.
❤️ Community
Join the Plane community on GitHub Discussions and our Discord server. We follow a Code of conduct in all our community channels.
Feel free to ask questions, report bugs, participate in discussions, share ideas, request features, or showcase your projects. We’d love to hear from you!
🛡️ Security
If you discover a security vulnerability in Plane, please report it responsibly instead of opening a public issue. We take all legitimate reports seriously and will investigate them promptly. See Security policy for more info.
To disclose any security issues, please email us at security@plane.so.
🤝 Contributing
There are many ways you can contribute to Plane:
- Report bugs or submit feature requests.
- Review the documentation and submit pull requests to improve it—whether it's fixing typos or adding new content.
- Talk or write about Plane or any other ecosystem integration and let us know!
- Show your support by upvoting popular feature requests.
Please read CONTRIBUTING.md for details on the process for submitting pull requests to us.
Repo activity
We couldn't have done this without you.
License
This project is licensed under the GNU Affero General Public License v3.0.