* Refactor and extend cleanup tasks for logs and versions
- Consolidate API log deletion into cleanup_task.py - Add tasks to
delete old email logs, page versions, and issue description versions -
Update Celery schedule and imports for new tasks
* chore: update cleanup task with mongo changes
* fix: update log deletion task name for clarity
* fix: enhance MongoDB archival error handling in cleanup task
- Added a parameter to check MongoDB availability in the flush_to_mongo_and_delete function.
- Implemented error logging for MongoDB archival failures.
- Updated calls to flush_to_mongo_and_delete to include the new parameter.
* fix: correct parameter name in cleanup task function call
- Updated the parameter name from 'mode' to 'model' in the process_cleanup_task function to ensure consistency and clarity in the code.
* fix: improve MongoDB connection parameter handling in MongoConnection class
- Replaced direct access to settings with getattr for MONGO_DB_URL and MONGO_DB_DATABASE to enhance robustness.
- Added warning logging for missing MongoDB connection parameters.
* chore: added teamspace permission for pages
* chore: updated the comment in permission layer
* chore: stored the teamspace member in variable
* chore: optimised the logic
* chore: updated the teamspace permission
* add missing fields and methods in endpoints
* add POST method for project members
* make project_id as uuid in url pattern
* remove post method
* fix method reordering
* fix: wrong WI count while scrolling
* chore: optimize issue queryset
* fix: use separate query for total_count_queryset
* fix: guest visibility constraint
* fix: use separate query for total_count_queryset in external api
* fix: use queryset.count()
* refactor: streamline database connection handling in AutomationConsumer and enhance outbox record formatting
* chore: update outbox cleaner schedule and cutoff days configuration
* feat: enhance database connection handling in AutomationConsumer
Added a method to ensure the database connection is alive and integrated it into message processing to improve reliability. This change replaces direct calls to close old connections with a more robust connection check.
* fix: handle permanent database connection loss in AutomationConsumer
Added a warning log and system exit for cases where the database connection is lost permanently, improving the robustness of the consumer's error handling.
* fix: improve error handling for database connection loss in AutomationConsumer
Added a warning log and system exit for permanent database connection loss, enhancing the consumer's resilience against connection issues.
* Refactor API logs task and outbox cleaner logic
- Use MongoConnection in API logs task - Add ENABLE_OUTBOX_POLLER
setting - Update outbox cleaner to respect poller setting - Add
cleanup_junk_data.py placeholder
* init: automations store and components
* chore: updated components
* refactor: correct semantics
* chore: modal components added
* chore: update conjunction translations
* Pgtrigger and Outbox prototype
* feat: add event stream app with models and migrations
- Introduced the event_stream app with initial models: Outbox and IssueProxy.
- Implemented a PostgreSQL trigger for logging updates to issues.
- Added necessary migrations and updated settings to include the new app.
* refactor: migrate event publisher to event stream
- Updated imports in test files to reflect the new event_stream app structure.
- Removed the event_publisher app, including its models, migrations, and tests.
- Introduced MemorySafeOutboxEventListener and related service for handling outbox events.
- Added management command for starting the outbox event listener with memory safety.
- Implemented PostgreSQL triggers for outbox notifications in the new event_stream app.
* refactor: enhance issue serializer and event stream models
- Updated IssueSerializer and IssueCreateSerializer to manage assignees and labels more efficiently by calculating additions and removals instead of deleting all and re-adding.
- Introduced new proxy models for CycleIssue and ModuleIssue with corresponding PostgreSQL triggers for event handling.
- Refactored Outbox model to include entity type and ID for better event tracking.
- Removed outdated IssueProxy model and its associated triggers, streamlining the event stream architecture.
* chore: update base requirements for OpenSearch and add pika for RabbitMQ
- Retained django-opensearch-dsl version 0.7.0 in requirements.
- Added pika version 1.3.2 for RabbitMQ integration.
* refactor: update import paths for event stream listener
- Changed import statements in test_competitive_listeners.py, service.py, and listen_outbox_events.py to reflect the new module structure by removing the 'apiserver' prefix.
* feat: implement PostgreSQL LISTEN/NOTIFY listener for outbox events
- Added DatabaseConnection class to manage psycopg3 connections for LISTEN/NOTIFY.
- Introduced NotificationListener class to handle PostgreSQL NOTIFY messages and dispatch them to registered handlers.
- Updated Command class to utilize NotificationListener for listening to outbox events, replacing the previous restartable listener implementation.
- Improved error handling and logging throughout the listener process.
* feat: enhance event processing with advisory locks
- Implemented advisory lock mechanism to prevent concurrent processing of events.
- Added methods to acquire and release advisory locks in the DatabaseConnection class.
- Updated root_handler to fetch complete outbox data and mark events as processed only if all handlers succeed.
- Improved error handling and logging for event processing.
* fix: improve event logging and error handling in NotificationListener
- Updated event logging to include event ID for better traceability.
- Enhanced error handling in the command's handle method to log exceptions and maintain robustness during event listening.
* feat: add memory monitoring and auto-restart functionality to event listener
- Introduced MemoryMonitor class to track memory usage and event processing counts.
- Implemented automatic restarts for the NotificationListener based on memory and event limits.
- Enhanced command line options to configure memory limits, event limits, and memory check intervals.
- Improved logging for memory stats and event processing to aid in monitoring and debugging.
* feat: update local and production settings for event logging
- Introduced logging configuration for 'plane.event_stream' in both local and production settings to enhance event logging capabilities.
* chore: Implement async outbox polling mechanism with memory monitoring and event handling. Add management command for outbox poller and create necessary database migrations for outbox triggers and indexes.
* docs: Add comprehensive README for Event Stream system detailing architecture, components, configuration, usage, and performance tuning.
* feat: Enhance MongoDB integration with a singleton connection manager and implement outbox cleanup task for migrating records from PostgreSQL to MongoDB
* feat: Add asyncio support for tests and new test files for outbox polling
* Merge branch 'event-stream' of github.com:makeplane/plane-ee into event-stream
* refactor: Update event types in outbox triggers for cycle, issue, and module models; add IssueComment, IssueLink, and IssueRelation proxies with corresponding triggers
* refactor: Update import paths for JsonFormatter and consolidate OpenFeature imports; add unit tests for OutboxPoller and related classes
* refactor: Replace all_objects with objects in Outbox queries and update test fixtures for outbox records
* feat: Add daily outbox cleaner task to Celery and update type hint for MongoDB collection
* refactor: Enhance memory monitoring and outbox polling logic with improved logging and delay handling
* feat: Implement outbox triggers for issue, issue attachment, issue link, and issue relation models with dynamic event type determination
* chore: Remove obsolete files related to multiple listeners and competitive processing tests
* Delete service.py
* feat: Add outbox poller script to manage database migrations and process events with configurable parameters
* refactor: Optimize EpicCreateSerializer to handle initiative, assignee, and label updates with conflict management and improved logic for adding/removing related entities
* feat: Update issue proxy triggers for outbox events
- Removed the existing 'issue_outbox_update' trigger from the IssueProxy model.
- Added a new 'issue_outbox_update' trigger with enhanced logic to handle conversion events when the issue type changes.
- The new trigger captures updates and soft deletes, inserting appropriate events into the outbox.
* feat: Refactor issue proxy triggers and add new outbox event handling
- Removed and replaced existing triggers for IssueProxy, IssueAssigneeProxy, IssueLabelProxy, CycleIssueProxy, and ModuleIssueProxy.
- Introduced new triggers to handle outbox events for various issue-related actions, including creation, updates, and deletions.
- Enhanced logic for event type determination based on issue type, ensuring accurate event handling in the outbox.
* refactor: Update serializers to use ID fields for assignees and labels
- Changed the handling of assignees and labels in IssueCreateSerializer and EpicCreateSerializer to use ID fields instead of object references.
- Improved logic for determining assignees and labels to add or remove by using lists of IDs.
- Enhanced code readability with comments explaining the changes.
* feat: Enhance issue event handling in outbox triggers
- Updated triggers for IssueProxy, IssueAssigneeProxy, and IssueLabelProxy to determine event types based on issue type, including handling for epics.
- Introduced filtered data for outbox events to exclude description fields, improving data integrity and reducing unnecessary payload size.
- Enhanced logic for detecting changes during updates, ensuring only relevant changes are captured and processed.
* feat: implement connection pooling for outbox poller
- Introduced DatabaseConnectionPool class to manage async connection pooling using psycopg_pool.
- Enhanced outbox polling logic to utilize connection pooling for improved performance and resource management.
- Added health check and statistics retrieval methods for the connection pool.
- Updated outbox model and migration to include claimed_at field for better event tracking.
- Refactored tests to validate new connection pooling functionality.
* feat: Introduce automation engine with models and registry
- Added core components for the automation engine, including models for Automation, AutomationVersion, AutomationNode, AutomationEdge, AutomationRun, and NodeExecution.
- Implemented a NodeRegistry for managing automation nodes and a decorator for easy registration.
- Created initial files for automation app configuration and documentation, including an entity-relationship diagram and detailed model explanations.
- Established a migration for the new automation models to integrate with the existing database schema.
* refactor: move MongoConnection to a new settings module and update imports
- Created a new mongo.py file to define the MongoConnection class for managing MongoDB connections.
- Updated the import path in outbox_cleaner.py to reference the new MongoConnection location.
* refactor: update import path for MongoConnection in outbox_cleaner.py
* chore: remove scout_apm from production settings and requirements
- Deleted scout_apm from the installed apps in production.py.
- Removed scout-apm dependency from base.txt requirements.
* feat: add claimed_at field to outbox events and update requirements
- Added claimed_at field to the outbox event model for enhanced event tracking.
- Updated outbox poller to handle claimed_at in event processing.
- Included django-pgtrigger in base.txt requirements for database trigger management.
* feat: enhance memory monitoring in outbox poller
- Updated MemoryMonitor to signal for restarts instead of exiting on memory limit exceedance.
- Added methods to check for restart requests and wait for restart signals.
- Improved outbox poller logic to handle memory monitoring more gracefully during processing.
- Ensured all claimed rows are processed before initiating a restart.
* feat: implement graceful shutdown handling in outbox poller
- Added GracefulShutdownHandler class to manage shutdown signals (SIGTERM, SIGINT, SIGQUIT).
- Integrated shutdown handling into the outbox poller to allow for graceful exits during processing.
- Updated polling logic to check for shutdown requests and clean up resources accordingly.
- Enhanced command help text to reflect new signal handling capabilities.
* simplify AutomationEdge model by removing label and condition fields, and update unique constraint
* Enhance outbox event handling by adding workspace and project IDs
- Updated the outbox cleaner to include `workspace_id` and `project_id` in the deletion process.
- Modified the outbox poller to handle new fields in event processing.
- Adjusted models and migrations to support the new fields in the outbox table.
- Updated tests to ensure proper handling of workspace and project IDs in outbox records.
* chore: folder migration
* fix: type errors
* Refactor AutomationNode model and migration
- Removed `node_id`, `position_x`, `position_y`, and `execution_order` fields from the AutomationNode model.
- Updated migration to reflect the removal of constraints and adjusted the ordering of the model.
- Cleaned up the migration dependencies for better clarity.
* Add automation serializers and endpoints
- Introduced AutomationWriteSerializer and AutomationReadSerializer for handling automation data.
- Added AutomationNodeWriteSerializer and AutomationNodeReadSerializer for automation node management.
- Created AutomationEndpoint and AutomationNodeEndpoint views to manage automation and automation nodes with CRUD operations.
- Implemented permission checks for different roles on the endpoints.
* Merge branch 'preview' of github.com:makeplane/plane-ee into event-stream
* Add automation scope choices and update serializers and views
- Introduced AutomationScopeChoices for defining automation scopes.
- Updated AutomationWriteSerializer and AutomationReadSerializer to include scope validation and new fields.
- Created automation URL patterns and endpoints for managing automations.
- Implemented feature flag checks for automation-related endpoints.
- Enhanced permission checks for CRUD operations on automations.
* Refactor automation models, serializers, and views for improved structure and functionality
- Updated migration file to enhance automation-related models, including Automation, AutomationVersion, AutomationRun, AutomationNode, and AutomationEdge.
- Refactored serializers to include additional fields and validation for AutomationNode.
- Introduced new endpoints for managing AutomationNodes, including CRUD operations with feature flag checks.
- Improved URL patterns for automation and automation node management.
- Enhanced permission checks for automation-related operations.
* Add AutomationEdge serializers and endpoints for managing automation edges
- Introduced AutomationEdgeWriteSerializer and AutomationEdgeReadSerializer for handling automation edge data.
- Created AutomationEdgeEndpoint to manage CRUD operations for automation edges with appropriate feature flag checks.
- Updated URL patterns to include routes for automation edges.
- Refactored existing automation node endpoint to inherit from AutomationBaseEndpoint for consistency.
* feat: add Automation Bot type and enhance automation model
- Introduced a new BotTypeEnum value for "AUTOMATION_BOT".
- Added a OneToOneField for bot_user in the Automation model to associate a bot user with automations.
- Created a new AutomationActivity model to track activities related to automations, including fields for action, old and new values, and related entities.
* feat: add automation models and constraints
- Introduced new models for Automation, AutomationNode, AutomationRun, NodeExecution, AutomationVersion, AutomationEdge, and AutomationActivity.
- Implemented foreign key relationships and constraints to ensure data integrity, including unique constraints with conditions for deleted records.
- Added indexes to optimize query performance on various fields across the new models.
- Enhanced existing models with additional fields and relationships to support automation functionality.
* feat: rich filters store architecture and UI components
* improvement: use map instead of object for automations store
* improvement: rename automations v2 components to automations
* feat: automation nodes and edges
* feat: enhance automation functionality with new models and endpoints
- Added AutomationRun and NodeExecution models to track automation runs and their execution details.
- Introduced AutomationDetailReadSerializer to provide detailed information about automations, including nodes and edges.
- Updated AutomationEndpoint to create and manage automation bots, enhancing automation capabilities.
- Created AutomationRunEndpoint for retrieving automation run data, supporting both single and multiple run queries.
- Refactored existing serializers and views to incorporate new features and improve overall structure.
* improvement: refactor automation nodes
* feat: publisher for pushing events from the poller to the queue
* feat: Implement RabbitMQ Event Stream Publisher and Outbox Event Model
- Added EventStreamPublisher class for reliable event publishing to RabbitMQ with support for fanout exchanges, message persistence, and concurrent access handling.
- Introduced OutboxEvent dataclass for structured representation of outbox events, including methods for conversion to various formats.
- Updated OutboxPoller to utilize the new publisher for handling outbox events, improving event processing and publishing logic.
* feat: Add logging for published messages in EventStreamPublisher
- Enhanced the EventStreamPublisher class to log published messages, including instance ID and message body for better traceability and debugging.
* feat: Enhance Outbox Poller Tests and Introduce Event Stream Publisher Tests
- Refactored test cases in `test_outbox_poller.py` to simplify row tuple creation and improve readability.
- Added new test suite for `EventStreamPublisher` in `test_publisher.py`, covering initialization, connection management, message publishing, error handling, and thread safety.
- Introduced a new `__init__.py` file in the event stream tests directory to support module structure.
---------
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
* feat: implement automation activity tracking functionality
- Added a new module for tracking automation activities, including creation, updates, and deletions.
- Introduced functions to handle different types of automation activities and log changes to relevant fields.
- Updated AutomationEndpoint to trigger automation activity tasks upon creation, update, and deletion of automations.
- Enhanced error handling and logging for automation activity processes.
* feat: expand automation activity tracking with node and edge functionalities
- Added functions to track creation, updates, and deletions of automation nodes and edges.
- Implemented new activity types for node and edge activities in the automation activity task.
- Updated AutomationNodeEndpoint and AutomationEdgeEndpoint to trigger activity tracking upon relevant actions.
- Enhanced serialization of requested data for activity logging, ensuring accurate tracking of changes.
* feat: add automation activity tracking for activities
- Introduced AutomationActivityReadSerializer to handle serialization of automation activity data.
- Created AutomationActivityEndpoint for managing CRUD operations related to automation activities.
- Updated URL patterns to include routes for automation activities.
- Enhanced existing serializers and views to support new activity tracking functionalities.
* feat: add unit tests for automation activity tracking
- Introduced comprehensive unit tests for automation activity functions, including creation, deletion, and updates of automation activities, nodes, and edges.
- Implemented test cases to validate the tracking of field changes and ensure accurate logging of activities.
- Enhanced test coverage for various scenarios, including valid and invalid data inputs, to ensure robustness of automation activity tracking functionalities.
* feat: Implement automation execution engine and node management
- Added AutomationExecutionEngine class to handle the execution of automation workflows, including validation of events, loading of nodes, and execution of triggers, conditions, and actions.
- Introduced various node types (triggers, conditions, actions) with specific implementations for handling events and performing actions within the automation framework.
- Enhanced the automation system with auto-discovery of node modules during Django startup, ensuring all nodes are registered and ready for execution.
- Created comprehensive README documentation for automation nodes, detailing available triggers, conditions, and their configurations.
- Added error handling and logging for improved traceability during automation execution.
- Implemented tests for the new automation features to ensure reliability and correctness.
This commit lays the foundation for a robust automation system capable of responding to database events and executing defined workflows.
* Added automations to installed apps
* refactor: remove unused import in automation activity test file
* chore: automations components restructure and API integration init
* improvement: add loader for automations list table
* fix: automations settings page layout
* fix: automation list loader
* improvement: reduce add node button height
* feat: update automation modal.
* refactor: simplify label ID handling in IssueCreateSerializer
* improvement: rename `actions` to `action-buttons` in automation details sidebar.
* feat: add automation status endpoint to toggle status for different automations
* feat: automation comment actions
* chore: default last run status to None
* chore: add total success count
* chore: add total failed count
* improvement: automations list table
* chore: automation feature flag update and minor fix in status endpoint
* fix: automation status endpoint feature flag
* feat: automation details page header
* chore: remove support for multiple trigger nodes in a single automation
* feat: update automation status handling to include is_enabled flag
* improvements: minor enhancements for rich filters
* added button configuration to add filters button
* added option to disable transition for filter item
* fixed filter item label truncation
* added support for changing work item filter config keys
* improvements: add filter button config initialization
* chore: minor type name updates
* Updated migration
* Implement automation consumer and management commands for event processing
- Added AutomationConsumer class to handle RabbitMQ events and dispatch to Celery tasks.
- Created management commands for running the automation consumer and creating automations interactively.
- Introduced Celery tasks for executing automation logic and cleaning up processed events.
- Updated settings to include automation-related configurations.
- Implemented logging for better traceability of automation processes.
* Enhance automation execution engine with structured results and multiple action support
- Introduced new data classes for standardized execution context and results: ExecutionContext, NodeResult, and AutomationResult.
- Updated AutomationExecutionEngine to support multiple action nodes executed sequentially, halting on the first failure.
- Refactored node execution methods to improve clarity and maintainability.
- Enhanced error handling and logging for better traceability during automation execution.
- Updated README to reflect changes in automation execution flow and node capabilities.
* feat: enhance automation model to create bot user and update field tracking
* feat: add initiator_id field to Outbox model and update migration dependencies
* feat: condition node for automations
* chore: enable feature flag check for automation create and update status endpoint
* feat: add initiator_id to Outbox insertions across event stream models
- Updated Outbox insert statements in various models to include initiator_id.
- Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure proper tracking of the user initiating changes.
- Enhanced event handling for soft deletes and regular updates to maintain consistency in data tracking.
* feat: enable / disable automations
* feat: include initiator_id in OutboxEvent model and database interactions
- Added initiator_id field to the OutboxEvent model for enhanced tracking.
- Updated database queries in outbox_poller.py to include initiator_id in insertions.
- Ensured consistency in data handling across event stream components.
* refactor: update handle_row function to use OutboxEvent model
- Changed the parameter type of handle_row from Dict to OutboxEvent for better type safety.
- Updated logging to utilize the to_dict method of OutboxEvent for consistent event data representation.
* chore: update `edgeIdsToDelete` to `edgeIdsToRemove`
* refactor: simplify EventStreamPublisher by removing default queue and message TTL parameters, and updating setup methods for exchange only
* feat: add change property action
* TODOs
* chore: automation activity
* chore: added issue for issue created trigger
* chore: updated migrations
* chore: added priority, date and state changes
* chore: update automation migration
* chore: remove label activity
* chore: created the comment activity
* fix: logical operators casing
* feat: add docker entrypoint script for automation consumer
- Introduced a new script to manage the automation consumer process.
- The script includes commands to wait for the database and migrations before running the automation consumer.
* feat: add automation models and migrations
- Introduced new models for Automation, AutomationActivity, AutomationEdge, AutomationNode, AutomationRun, AutomationVersion, NodeExecution, and ProcessedAutomationEvent.
- Implemented necessary migrations to support the new automation features, including relationships and constraints.
- Enhanced event handling capabilities with detailed tracking and management of automation processes.
* chore: trigger main content condition preview
* chore: update condition preview limit
* chore: validated conditions for automations
* chore: added check for automation and import job
* refactor: streamline event processing by ignoring system-originated events
* fix: automation activities
* refactor: update SQL command for setting initiator type in automation and import tasks
* chore: update lxml and xmlsec versions in base requirements
* chore: update outbox cleaner cutoff days to 7 and include it in settings
* fix: update SQL command for setting initiator type in data import and bulk update tasks
* feat: add triggers for issue assignee and label management in outbox
- Implemented triggers for inserting and updating issue assignee and label data in the outbox.
- Adjusted SQL logic to handle previous attributes correctly during assignee and label updates.
- Changed trigger execution timing from AFTER to BEFORE for better handling of soft deletes.
* feat: enhance automation consumer script with queue and prefetch options
- Added support for specifying the automation events queue and prefetch count in the docker entrypoint script for the automation consumer.
* fix: update automation consumer script and event stream exchange name
- Changed the queue name in the automation consumer script to use the new environment variable for event stream.
- Updated the default exchange name in the event stream publisher to align with the new naming convention.
* refactor: update automation consumer settings and permissions
- Changed the queue name and exchange name in the automation consumer and related scripts to use new environment variables.
- Updated file permissions for the docker entrypoint script to ensure proper execution.
* chore: activity filters
* fix: activity filters
* chore: automation activity description
* chore: added new fields in automation run
* improvement: add custom automations to project settings page
* improvement: minor enhancements and fixes.
* feat: update automation consumer settings and logging configuration
* Changed default queue name for automation events in run_automation_consumer.py.
* Added logging configuration for plane.automations.consumer in both local and production settings.
* Removed unused node_execution.py file.
* minor cleanup
* chore: minor improvements
* feat: update automation execution tracking and logging
* Added logic to update the last run timestamp and increment the run count for automations.
* Enhanced the field tracking in automation activity logging to include node type for better clarity.
---------
Co-authored-by: Aaryan Khandelwal <aaryankhandu123@gmail.com>
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
Co-authored-by: pablohashescobar <nikhilschacko@gmail.com>
Co-authored-by: Nikhil <118773738+pablohashescobar@users.noreply.github.com>
Co-authored-by: NarayanBavisetti <narayan3119@gmail.com>
Co-authored-by: Bavisetti Narayan <72156168+NarayanBavisetti@users.noreply.github.com>
* feat: add list endpoint for listing all values related to all properties for a workitem
* feat: enhance issue property value retrieval by including issue type filtering
* refactor: clean up issue property validators and improve API documentation for issue property value endpoints
* improvement: update empty state assets for recurring work items
* improvement: add all missing recurring work item activity types
* fix: update recurring work items submitting state
* chore: update upgrade empty state
* feat: add all missing translations
* add external apis for epic
* epic in plane api sdk
* add tests for epic endpoints + support epics in issue-identifier endpoint
* format files
* use choices from model + fixes
---------
Co-authored-by: sriramveeraghanta <veeraghanta.sriram@gmail.com>
