* Pgtrigger and Outbox prototype
* feat: add event stream app with models and migrations
- Introduced the event_stream app with initial models: Outbox and IssueProxy.
- Implemented a PostgreSQL trigger for logging updates to issues.
- Added necessary migrations and updated settings to include the new app.
* refactor: migrate event publisher to event stream
- Updated imports in test files to reflect the new event_stream app structure.
- Removed the event_publisher app, including its models, migrations, and tests.
- Introduced MemorySafeOutboxEventListener and related service for handling outbox events.
- Added management command for starting the outbox event listener with memory safety.
- Implemented PostgreSQL triggers for outbox notifications in the new event_stream app.
* refactor: enhance issue serializer and event stream models
- Updated IssueSerializer and IssueCreateSerializer to manage assignees and labels more efficiently by calculating additions and removals instead of deleting all and re-adding.
- Introduced new proxy models for CycleIssue and ModuleIssue with corresponding PostgreSQL triggers for event handling.
- Refactored Outbox model to include entity type and ID for better event tracking.
- Removed outdated IssueProxy model and its associated triggers, streamlining the event stream architecture.
* chore: update base requirements for OpenSearch and add pika for RabbitMQ
- Retained django-opensearch-dsl version 0.7.0 in requirements.
- Added pika version 1.3.2 for RabbitMQ integration.
* refactor: update import paths for event stream listener
- Changed import statements in test_competitive_listeners.py, service.py, and listen_outbox_events.py to reflect the new module structure by removing the 'apiserver' prefix.
* feat: implement PostgreSQL LISTEN/NOTIFY listener for outbox events
- Added DatabaseConnection class to manage psycopg3 connections for LISTEN/NOTIFY.
- Introduced NotificationListener class to handle PostgreSQL NOTIFY messages and dispatch them to registered handlers.
- Updated Command class to utilize NotificationListener for listening to outbox events, replacing the previous restartable listener implementation.
- Improved error handling and logging throughout the listener process.
* feat: enhance event processing with advisory locks
- Implemented advisory lock mechanism to prevent concurrent processing of events.
- Added methods to acquire and release advisory locks in the DatabaseConnection class.
- Updated root_handler to fetch complete outbox data and mark events as processed only if all handlers succeed.
- Improved error handling and logging for event processing.
* fix: improve event logging and error handling in NotificationListener
- Updated event logging to include event ID for better traceability.
- Enhanced error handling in the command's handle method to log exceptions and maintain robustness during event listening.
* feat: add memory monitoring and auto-restart functionality to event listener
- Introduced MemoryMonitor class to track memory usage and event processing counts.
- Implemented automatic restarts for the NotificationListener based on memory and event limits.
- Enhanced command line options to configure memory limits, event limits, and memory check intervals.
- Improved logging for memory stats and event processing to aid in monitoring and debugging.
* feat: update local and production settings for event logging
- Introduced logging configuration for 'plane.event_stream' in both local and production settings to enhance event logging capabilities.
* chore: Implement async outbox polling mechanism with memory monitoring and event handling. Add management command for outbox poller and create necessary database migrations for outbox triggers and indexes.
* docs: Add comprehensive README for Event Stream system detailing architecture, components, configuration, usage, and performance tuning.
* feat: Enhance MongoDB integration with a singleton connection manager and implement outbox cleanup task for migrating records from PostgreSQL to MongoDB
* feat: Add asyncio support for tests and new test files for outbox polling
* Merge branch 'event-stream' of github.com:makeplane/plane-ee into event-stream
* refactor: Update event types in outbox triggers for cycle, issue, and module models; add IssueComment, IssueLink, and IssueRelation proxies with corresponding triggers
* refactor: Update import paths for JsonFormatter and consolidate OpenFeature imports; add unit tests for OutboxPoller and related classes
* refactor: Replace all_objects with objects in Outbox queries and update test fixtures for outbox records
* feat: Add daily outbox cleaner task to Celery and update type hint for MongoDB collection
* refactor: Enhance memory monitoring and outbox polling logic with improved logging and delay handling
* feat: Implement outbox triggers for issue, issue attachment, issue link, and issue relation models with dynamic event type determination
* chore: Remove obsolete files related to multiple listeners and competitive processing tests
* Delete service.py
* feat: Add outbox poller script to manage database migrations and process events with configurable parameters
* refactor: Optimize EpicCreateSerializer to handle initiative, assignee, and label updates with conflict management and improved logic for adding/removing related entities
* feat: Update issue proxy triggers for outbox events
- Removed the existing 'issue_outbox_update' trigger from the IssueProxy model.
- Added a new 'issue_outbox_update' trigger with enhanced logic to handle conversion events when the issue type changes.
- The new trigger captures updates and soft deletes, inserting appropriate events into the outbox.
* feat: Refactor issue proxy triggers and add new outbox event handling
- Removed and replaced existing triggers for IssueProxy, IssueAssigneeProxy, IssueLabelProxy, CycleIssueProxy, and ModuleIssueProxy.
- Introduced new triggers to handle outbox events for various issue-related actions, including creation, updates, and deletions.
- Enhanced logic for event type determination based on issue type, ensuring accurate event handling in the outbox.
* refactor: Update serializers to use ID fields for assignees and labels
- Changed the handling of assignees and labels in IssueCreateSerializer and EpicCreateSerializer to use ID fields instead of object references.
- Improved logic for determining assignees and labels to add or remove by using lists of IDs.
- Enhanced code readability with comments explaining the changes.
* feat: Enhance issue event handling in outbox triggers
- Updated triggers for IssueProxy, IssueAssigneeProxy, and IssueLabelProxy to determine event types based on issue type, including handling for epics.
- Introduced filtered data for outbox events to exclude description fields, improving data integrity and reducing unnecessary payload size.
- Enhanced logic for detecting changes during updates, ensuring only relevant changes are captured and processed.
* feat: implement connection pooling for outbox poller
- Introduced DatabaseConnectionPool class to manage async connection pooling using psycopg_pool.
- Enhanced outbox polling logic to utilize connection pooling for improved performance and resource management.
- Added health check and statistics retrieval methods for the connection pool.
- Updated outbox model and migration to include claimed_at field for better event tracking.
- Refactored tests to validate new connection pooling functionality.
* refactor: move MongoConnection to a new settings module and update imports
- Created a new mongo.py file to define the MongoConnection class for managing MongoDB connections.
- Updated the import path in outbox_cleaner.py to reference the new MongoConnection location.
* refactor: update import path for MongoConnection in outbox_cleaner.py
* chore: remove scout_apm from production settings and requirements
- Deleted scout_apm from the installed apps in production.py.
- Removed scout-apm dependency from base.txt requirements.
* feat: add claimed_at field to outbox events and update requirements
- Added claimed_at field to the outbox event model for enhanced event tracking.
- Updated outbox poller to handle claimed_at in event processing.
- Included django-pgtrigger in base.txt requirements for database trigger management.
* feat: enhance memory monitoring in outbox poller
- Updated MemoryMonitor to signal for restarts instead of exiting on memory limit exceedance.
- Added methods to check for restart requests and wait for restart signals.
- Improved outbox poller logic to handle memory monitoring more gracefully during processing.
- Ensured all claimed rows are processed before initiating a restart.
* feat: implement graceful shutdown handling in outbox poller
- Added GracefulShutdownHandler class to manage shutdown signals (SIGTERM, SIGINT, SIGQUIT).
- Integrated shutdown handling into the outbox poller to allow for graceful exits during processing.
- Updated polling logic to check for shutdown requests and clean up resources accordingly.
- Enhanced command help text to reflect new signal handling capabilities.
* Enhance outbox event handling by adding workspace and project IDs
- Updated the outbox cleaner to include `workspace_id` and `project_id` in the deletion process.
- Modified the outbox poller to handle new fields in event processing.
- Adjusted models and migrations to support the new fields in the outbox table.
- Updated tests to ensure proper handling of workspace and project IDs in outbox records.
* Merge branch 'preview' of github.com:makeplane/plane-ee into event-stream
* feat: add initiator_id field to Outbox model and update migration dependencies
* feat: add initiator_id to Outbox insertions across event stream models
- Updated Outbox insert statements in various models to include initiator_id.
- Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure proper tracking of the user initiating changes.
- Enhanced event handling for soft deletes and regular updates to maintain consistency in data tracking.
* feat: include initiator_id in OutboxEvent model and database interactions
- Added initiator_id field to the OutboxEvent model for enhanced tracking.
- Updated database queries in outbox_poller.py to include initiator_id in insertions.
- Ensured consistency in data handling across event stream components.
* refactor: update handle_row function to use OutboxEvent model
- Changed the parameter type of handle_row from Dict to OutboxEvent for better type safety.
- Updated logging to utilize the to_dict method of OutboxEvent for consistent event data representation.
* feat: enhance Outbox model with initiator type and update related logic
- Added initiator_type field to the Outbox model to track the type of event initiator.
- Introduced InitiatorTypes enum for better clarity and management of initiator types.
- Updated OutboxEvent model and related methods to include initiator_type for consistent event data representation.
- Adjusted database interactions across event stream models to accommodate the new initiator_type field.
* feat: add initiator_type to Outbox insertions across event stream models
- Updated Outbox insert statements in various models to include initiator_type for better tracking of event initiators.
- Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure consistent handling of initiator_type during event processing.
- Enhanced event handling for both soft deletes and regular updates to maintain data integrity and tracking.
* feat: update event stream triggers to use After timing for improved consistency
- Changed trigger timing from Before to After for various event stream models including CycleIssueProxy, IssueProxy, and IssueAssigneeProxy to ensure that outbox updates reflect the final state of the entities.
- Enhanced logic in triggers to include previous attributes for better tracking of changes during updates.
- Adjusted related logic in IssueLabelProxy and IssueCommentProxy to maintain consistency across event handling.
* feat: enhance bulk update task with transaction management and initiator type
- Wrapped bulk creation of issue relations and parent ID updates in a transaction to ensure atomicity.
- Set the initiator type to 'SYSTEM.IMPORT' for both bulk creation and updates to improve tracking of operations.
- Improved database interaction by using a connection cursor for executing SQL commands.
---------
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
* chore: crud operations for recurring work item
* chore: added perodic task for recurring work item
* chore: removed the error statement
* fix: bg task for labels
* chore: added recurring work item model
* chore: added null field for recurring work item log
* feat: recurring work items UI
* chore: added activities for recurring task
* feat: add interval type to recurring work item form
* chore: updated the cront expression
* chore: minor updates for recurring work item serializer and interval type
* chore: removed the seconds calculation
* chore: added task fk to activity table
* improvement: interval dropdown width
* fix: start date validation
* chore: added created at for activity
* feat: add recurring work item activity logic
* chore: updated the migration file
* chore: update the activity for recursive task
* chore: deleteing the recurring task permanantely
* improvement: recurring work item activity helpers
* fix: recurring work item custom properties data preloading
* improvement: minor UI enhancements
* improvement: recurring work item activity helpers
* fix: init loader logic
* Use the right field namewhile creating activity
* chore: add hourly and minutly interval types for testing
* Added test intervals for debugging
* chore: update the periodic task to set null
* chore: added activity for recurring work item properties
* chore: updated the description activity
* chore: removed the fields
* chore: updated the type activity
* chore: added property in the model
* remove temporary minutly and hourly interval types from recurring work items
---------
Co-authored-by: Prateek Shourya <prateekshourya29@gmail.com>
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
* improvement: remove `issues_exists` property from work item types.
* Optimizations for issue type and epic type endpoints
---------
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
* added PageBinaryUpdateSerializer for binary data validation and update
* chore: added validation for description
* chore: removed the duplicated file
* Fixed coderabbit comments
- Improve content validation by consolidating patterns and enhancing recursion checks
- Updated `PageBinaryUpdateSerializer` to simplify assignment of validated data.
- Enhanced `content_validator.py` with consolidated dangerous patterns and added recursion depth checks to prevent stack overflow during validation.
- Improved readability and maintainability of validation functions by using constants for patterns.
---------
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
* chore: added comment
* feat: enhance IssueCommentDocument and search functionality
- Added new fields: issue_title, issue_sequence_id, and issue_type_id to IssueCommentDocument.
- Updated IssueCommentSearchSerializer to include the new fields and changed comment to a SerializerMethodField for better handling of highlighted comments.
- Modified EnhancedGlobalSearchEndpoint to retrieve the new fields and implement highlighting for comments in search results.
* Update IssueCommentDocument and serializers for improved search functionality
- Enhanced IssueCommentDocument by adding an analyzer to the comment field and renaming issue_title to issue_name.
- Updated multiple serializers to make fields like type_id and logo_props optional.
- Improved IssueCommentSearchSerializer to handle highlights more effectively and renamed issue_title to issue_name.
- Adjusted EnhancedGlobalSearchEndpoint to reflect the changes in field names and modified comment highlighting settings.
* feat: integrated apis for wroksapce-search
---------
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
* feat: restore background color and goals fields in Profile and Workspace models in migration 0099
* chore: update migration dependency to reflect recent changes in profile model migrations
* feat: enhance workitem template endpoints with feature flag and detailed retrieval
- Added feature flag check for the SubWorkitemTemplateEndpoint.
- Updated WorkitemProjectTemplateEndpoint to support fetching a specific template by primary key (pk) with prefetching of related workitem templates.
* feat: add 404 error handling for missing templates in WorkitemProjectTemplateEndpoint
* refactor: oauth flow to move app installation out of create token + get installation from token
* added tests for app installation flow and webhook fix
* replace app_installation_id with id as query param
- Added support for custom Q objects in OpenSearchHelper to allow complex filtering logic.
- Updated the EnhancedGlobalSearchEndpoint to utilize the new custom filter for page access control.
- Improved filter handling in the execute_multi_search method for better result organization.
