6 Commits

Author	SHA1	Message	Date
Nikhil	eef03e671e	[WEB-4238] feat: event stream implementation (#3684) ... * Pgtrigger and Outbox prototype * feat: add event stream app with models and migrations - Introduced the event_stream app with initial models: Outbox and IssueProxy. - Implemented a PostgreSQL trigger for logging updates to issues. - Added necessary migrations and updated settings to include the new app. * refactor: migrate event publisher to event stream - Updated imports in test files to reflect the new event_stream app structure. - Removed the event_publisher app, including its models, migrations, and tests. - Introduced MemorySafeOutboxEventListener and related service for handling outbox events. - Added management command for starting the outbox event listener with memory safety. - Implemented PostgreSQL triggers for outbox notifications in the new event_stream app. * refactor: enhance issue serializer and event stream models - Updated IssueSerializer and IssueCreateSerializer to manage assignees and labels more efficiently by calculating additions and removals instead of deleting all and re-adding. - Introduced new proxy models for CycleIssue and ModuleIssue with corresponding PostgreSQL triggers for event handling. - Refactored Outbox model to include entity type and ID for better event tracking. - Removed outdated IssueProxy model and its associated triggers, streamlining the event stream architecture. * chore: update base requirements for OpenSearch and add pika for RabbitMQ - Retained django-opensearch-dsl version 0.7.0 in requirements. - Added pika version 1.3.2 for RabbitMQ integration. * refactor: update import paths for event stream listener - Changed import statements in test_competitive_listeners.py, service.py, and listen_outbox_events.py to reflect the new module structure by removing the 'apiserver' prefix. * feat: implement PostgreSQL LISTEN/NOTIFY listener for outbox events - Added DatabaseConnection class to manage psycopg3 connections for LISTEN/NOTIFY. - Introduced NotificationListener class to handle PostgreSQL NOTIFY messages and dispatch them to registered handlers. - Updated Command class to utilize NotificationListener for listening to outbox events, replacing the previous restartable listener implementation. - Improved error handling and logging throughout the listener process. * feat: enhance event processing with advisory locks - Implemented advisory lock mechanism to prevent concurrent processing of events. - Added methods to acquire and release advisory locks in the DatabaseConnection class. - Updated root_handler to fetch complete outbox data and mark events as processed only if all handlers succeed. - Improved error handling and logging for event processing. * fix: improve event logging and error handling in NotificationListener - Updated event logging to include event ID for better traceability. - Enhanced error handling in the command's handle method to log exceptions and maintain robustness during event listening. * feat: add memory monitoring and auto-restart functionality to event listener - Introduced MemoryMonitor class to track memory usage and event processing counts. - Implemented automatic restarts for the NotificationListener based on memory and event limits. - Enhanced command line options to configure memory limits, event limits, and memory check intervals. - Improved logging for memory stats and event processing to aid in monitoring and debugging. * feat: update local and production settings for event logging - Introduced logging configuration for 'plane.event_stream' in both local and production settings to enhance event logging capabilities. * chore: Implement async outbox polling mechanism with memory monitoring and event handling. Add management command for outbox poller and create necessary database migrations for outbox triggers and indexes. * docs: Add comprehensive README for Event Stream system detailing architecture, components, configuration, usage, and performance tuning. * feat: Enhance MongoDB integration with a singleton connection manager and implement outbox cleanup task for migrating records from PostgreSQL to MongoDB * feat: Add asyncio support for tests and new test files for outbox polling * Merge branch 'event-stream' of github.com:makeplane/plane-ee into event-stream * refactor: Update event types in outbox triggers for cycle, issue, and module models; add IssueComment, IssueLink, and IssueRelation proxies with corresponding triggers * refactor: Update import paths for JsonFormatter and consolidate OpenFeature imports; add unit tests for OutboxPoller and related classes * refactor: Replace all_objects with objects in Outbox queries and update test fixtures for outbox records * feat: Add daily outbox cleaner task to Celery and update type hint for MongoDB collection * refactor: Enhance memory monitoring and outbox polling logic with improved logging and delay handling * feat: Implement outbox triggers for issue, issue attachment, issue link, and issue relation models with dynamic event type determination * chore: Remove obsolete files related to multiple listeners and competitive processing tests * Delete service.py * feat: Add outbox poller script to manage database migrations and process events with configurable parameters * refactor: Optimize EpicCreateSerializer to handle initiative, assignee, and label updates with conflict management and improved logic for adding/removing related entities * feat: Update issue proxy triggers for outbox events - Removed the existing 'issue_outbox_update' trigger from the IssueProxy model. - Added a new 'issue_outbox_update' trigger with enhanced logic to handle conversion events when the issue type changes. - The new trigger captures updates and soft deletes, inserting appropriate events into the outbox. * feat: Refactor issue proxy triggers and add new outbox event handling - Removed and replaced existing triggers for IssueProxy, IssueAssigneeProxy, IssueLabelProxy, CycleIssueProxy, and ModuleIssueProxy. - Introduced new triggers to handle outbox events for various issue-related actions, including creation, updates, and deletions. - Enhanced logic for event type determination based on issue type, ensuring accurate event handling in the outbox. * refactor: Update serializers to use ID fields for assignees and labels - Changed the handling of assignees and labels in IssueCreateSerializer and EpicCreateSerializer to use ID fields instead of object references. - Improved logic for determining assignees and labels to add or remove by using lists of IDs. - Enhanced code readability with comments explaining the changes. * feat: Enhance issue event handling in outbox triggers - Updated triggers for IssueProxy, IssueAssigneeProxy, and IssueLabelProxy to determine event types based on issue type, including handling for epics. - Introduced filtered data for outbox events to exclude description fields, improving data integrity and reducing unnecessary payload size. - Enhanced logic for detecting changes during updates, ensuring only relevant changes are captured and processed. * feat: implement connection pooling for outbox poller - Introduced DatabaseConnectionPool class to manage async connection pooling using psycopg_pool. - Enhanced outbox polling logic to utilize connection pooling for improved performance and resource management. - Added health check and statistics retrieval methods for the connection pool. - Updated outbox model and migration to include claimed_at field for better event tracking. - Refactored tests to validate new connection pooling functionality. * refactor: move MongoConnection to a new settings module and update imports - Created a new mongo.py file to define the MongoConnection class for managing MongoDB connections. - Updated the import path in outbox_cleaner.py to reference the new MongoConnection location. * refactor: update import path for MongoConnection in outbox_cleaner.py * chore: remove scout_apm from production settings and requirements - Deleted scout_apm from the installed apps in production.py. - Removed scout-apm dependency from base.txt requirements. * feat: add claimed_at field to outbox events and update requirements - Added claimed_at field to the outbox event model for enhanced event tracking. - Updated outbox poller to handle claimed_at in event processing. - Included django-pgtrigger in base.txt requirements for database trigger management. * feat: enhance memory monitoring in outbox poller - Updated MemoryMonitor to signal for restarts instead of exiting on memory limit exceedance. - Added methods to check for restart requests and wait for restart signals. - Improved outbox poller logic to handle memory monitoring more gracefully during processing. - Ensured all claimed rows are processed before initiating a restart. * feat: implement graceful shutdown handling in outbox poller - Added GracefulShutdownHandler class to manage shutdown signals (SIGTERM, SIGINT, SIGQUIT). - Integrated shutdown handling into the outbox poller to allow for graceful exits during processing. - Updated polling logic to check for shutdown requests and clean up resources accordingly. - Enhanced command help text to reflect new signal handling capabilities. * Enhance outbox event handling by adding workspace and project IDs - Updated the outbox cleaner to include `workspace_id` and `project_id` in the deletion process. - Modified the outbox poller to handle new fields in event processing. - Adjusted models and migrations to support the new fields in the outbox table. - Updated tests to ensure proper handling of workspace and project IDs in outbox records. * Merge branch 'preview' of github.com:makeplane/plane-ee into event-stream * feat: add initiator_id field to Outbox model and update migration dependencies * feat: add initiator_id to Outbox insertions across event stream models - Updated Outbox insert statements in various models to include initiator_id. - Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure proper tracking of the user initiating changes. - Enhanced event handling for soft deletes and regular updates to maintain consistency in data tracking. * feat: include initiator_id in OutboxEvent model and database interactions - Added initiator_id field to the OutboxEvent model for enhanced tracking. - Updated database queries in outbox_poller.py to include initiator_id in insertions. - Ensured consistency in data handling across event stream components. * refactor: update handle_row function to use OutboxEvent model - Changed the parameter type of handle_row from Dict to OutboxEvent for better type safety. - Updated logging to utilize the to_dict method of OutboxEvent for consistent event data representation. * feat: enhance Outbox model with initiator type and update related logic - Added initiator_type field to the Outbox model to track the type of event initiator. - Introduced InitiatorTypes enum for better clarity and management of initiator types. - Updated OutboxEvent model and related methods to include initiator_type for consistent event data representation. - Adjusted database interactions across event stream models to accommodate the new initiator_type field. * feat: add initiator_type to Outbox insertions across event stream models - Updated Outbox insert statements in various models to include initiator_type for better tracking of event initiators. - Adjusted related logic in CycleIssueProxy, IssueProxy, ModuleIssueProxy, and others to ensure consistent handling of initiator_type during event processing. - Enhanced event handling for both soft deletes and regular updates to maintain data integrity and tracking. * feat: update event stream triggers to use After timing for improved consistency - Changed trigger timing from Before to After for various event stream models including CycleIssueProxy, IssueProxy, and IssueAssigneeProxy to ensure that outbox updates reflect the final state of the entities. - Enhanced logic in triggers to include previous attributes for better tracking of changes during updates. - Adjusted related logic in IssueLabelProxy and IssueCommentProxy to maintain consistency across event handling. * feat: enhance bulk update task with transaction management and initiator type - Wrapped bulk creation of issue relations and parent ID updates in a transaction to ensure atomicity. - Set the initiator type to 'SYSTEM.IMPORT' for both bulk creation and updates to improve tracking of operations. - Improved database interaction by using a connection cursor for executing SQL commands. --------- Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>	2025-08-08 17:33:05 +05:30
Dheeraj Kumar Ketireddy	03a9eeb25b	[WEB-4140] OpenAPI spec and external API refactor (#3750) ... * Basic setup for drf-spectacular * Updated to only handle /api/v1 endpoints * feat: add asset and user endpoints with URL routing - Introduced new asset-related endpoints for user assets and server assets, allowing for asset uploads and management. - Added user endpoint to retrieve current user information. - Updated URL routing to include new asset and user patterns. - Enhanced issue handling with a new search endpoint for issues across multiple fields. - Expanded member management with a new endpoint for workspace members. * Group endpoints by tags * Detailed schema definitions and examples for asset endpoints * Removed unnecessary extension * Specify avatar_url field separately * chore: add project docs * chore: correct all errors * chore: added open spec in work items * feat: enhance cycle API endpoints with detailed OpenAPI specifications - Updated CycleAPIEndpoint and CycleIssueAPIEndpoint to include detailed OpenAPI schema definitions for GET, POST, PATCH, and DELETE operations. - Specified allowed HTTP methods for each endpoint in the URL routing. - Improved documentation for cycle creation, updating, and deletion, including request and response examples. * chore: added open spec in labels * chore: work item properties * feat: enhance API endpoints with OpenAPI specifications and HTTP method definitions - Added detailed OpenAPI schema definitions for various API endpoints including Intake, Module, and State. - Specified allowed HTTP methods for each endpoint in the URL routing for better clarity and documentation. - Improved request and response examples for better understanding of API usage. - Introduced unarchive functionality for cycles and modules with appropriate endpoint definitions. * chore: run formatter * Removed unnecessary settings for authentication * Refactors OpenAPI documentation structure Improves the organization and maintainability of the OpenAPI documentation by modularizing the `openapi_spec_helpers.py` file. The changes include: - Migrates common parameters, responses, examples, and authentication extensions to separate modules. - Introduces helper decorators for different endpoint types. - Updates view imports to use the new module paths. - Removes the legacy `openapi_spec_helpers.py` file. This refactoring results in a more structured and easier-to-maintain OpenAPI documentation setup. * Refactor OpenAPI endpoint specifications - Removed unnecessary parameters from the OpenAPI documentation for various endpoints in the asset, cycle, and project views. - Updated request structures to improve clarity and consistency across the API documentation. - Enhanced response formatting for better readability and maintainability. * Enhance API documentation with detailed endpoint descriptions Updated various API endpoints across the application to include comprehensive docstrings that clarify their functionality. Each endpoint now features a summary and detailed description, improving the overall understanding of their purpose and usage. This change enhances the OpenAPI specifications for better developer experience and documentation clarity. * Enhance API serializers and views with new request structures - Added new serializers for handling cycle and module issue requests, including `CycleIssueRequestSerializer`, `TransferCycleIssueRequestSerializer`, `ModuleIssueRequestSerializer`, and intake issue creation/updating serializers. - Updated existing serializers to improve clarity and maintainability, including the `UserAssetUploadSerializer` and `IssueAttachmentUploadSerializer`. - Refactored API views to utilize the new serializers, enhancing the request handling for cycle and intake issue endpoints. - Improved OpenAPI documentation by replacing inline request definitions with serializer references for better consistency and readability. * Refactor OpenAPI documentation and endpoint specifications - Replaced inline schema definitions with dedicated decorators for various endpoint types, enhancing clarity and maintainability. - Updated API views to utilize new decorators for user, cycle, intake, module, and project endpoints, improving consistency in OpenAPI documentation. - Removed unnecessary parameters and responses from endpoint specifications, streamlining the documentation for better readability. - Enhanced the organization of OpenAPI documentation by modularizing endpoint-specific decorators and parameters. * chore: correct formatting * chore: correct formatting for all api folder files * refactor: clean up serializer imports and test setup - Removed unused `StateLiteSerializer` import from the serializer module. - Updated test setup to include a noqa comment for the `django_db_setup` fixture, ensuring clarity in the code. - Added missing commas in user data dictionary for consistency. * feat: add project creation and update serializers with validation - Introduced `ProjectCreateSerializer` and `ProjectUpdateSerializer` to handle project creation and updates, respectively. - Implemented validation to ensure project leads and default assignees are members of the workspace. - Updated API views to utilize the new serializers for creating and updating projects, enhancing request handling. - Added OpenAPI documentation references for the new serializers in the project API endpoints. * feat: update serializers to include additional read-only fields * refactor: rename intake issue serializers and enhance structure - Renamed `CreateIntakeIssueRequestSerializer` to `IntakeIssueCreateSerializer` and `UpdateIntakeIssueRequestSerializer` to `IntakeIssueUpdateSerializer` for clarity. - Introduced `IssueSerializer` for nested issue data in intake requests, improving the organization of serializer logic. - Updated API views to utilize the new serializer names, ensuring consistency across the codebase. * refactor: rename issue serializer for intake and enhance API documentation - Renamed `IssueSerializer` to `IssueForIntakeSerializer` for better clarity in the context of intake issues. - Updated references in `IntakeIssueCreateSerializer` and `IntakeIssueUpdateSerializer` to use the new `IssueForIntakeSerializer`. - Added OpenAPI documentation for the `get_workspace_work_item` endpoint, detailing parameters and responses for improved clarity. * chore: modules and cycles serializers * feat: add new serializers for label and issue link management - Introduced `LabelCreateUpdateSerializer`, `IssueLinkCreateSerializer`, `IssueLinkUpdateSerializer`, and `IssueCommentCreateSerializer` to enhance the handling of label and issue link data. - Updated existing API views to utilize the new serializers for creating and updating labels, issue links, and comments, improving request handling and validation. - Added `IssueSearchSerializer` for searching issues, streamlining the search functionality in the API. * Don't consider read only fields as required * Add setting to separate request and response definitions * Added OAuth 2 authentication for OpenAPI Spec * Fixed avatar_url warning on openapi spec generation * Fixed errors with openapi spec generation * Enhance OpenAPI documentation for issue types and properties - Updated issue type and property API endpoints to include detailed OpenAPI documentation using decorators. - Added support for various HTTP methods (GET, POST, PATCH, DELETE) in the issue type and property views. - Introduced new parameters and response schemas to improve API clarity and usability. - Updated settings to categorize endpoints under "Work Item Types" and "Work Item Properties" for better organization in the OpenAPI spec. * Made spectacular disabled by default * Moved spectacular settings into separate file and added detailed descriptions to tags * Specify methods for asset urls * Better server names * Added filter to exclude server and page endpoints from API v1 paths in OpenAPI hooks * Refine API endpoint methods and enhance OpenAPI documentation for worklogs - Specified HTTP methods for worklog API endpoints to support POST, GET, PATCH, and DELETE operations. - Added detailed descriptions for "Work Item Worklogs," "Work Item Types," and "Work Item Properties" in OpenAPI settings to improve documentation clarity and usability. * Refactor OpenAPI decorators for issue property endpoints - Updated decorators for issue property options and values to improve documentation clarity. - Introduced new `PROPERTY_ID_PARAMETER` for better parameter management in API paths. - Removed redundant parameters from issue worklog API methods to streamline the documentation. * Enhance API documentation with summaries for various endpoints - Added summary descriptions for user asset, cycle, intake, issue, member, module, project, state, and user API endpoints to improve clarity and usability of the API documentation. - Updated the OpenAPI specifications to reflect these changes, ensuring better understanding for developers interacting with the API. * Add contact information to OpenAPI settings - Included contact details for Plane in the OpenAPI settings to enhance API documentation and provide developers with a direct point of contact for support. - This addition aims to improve the overall usability and accessibility of the API documentation. * Enhance OpenAPI documentation across multiple endpoints - Added summaries to various API endpoints for improved clarity and usability. - Updated decorators for user, asset, cycle, intake, issue type, issue, module, project, state, and worklog endpoints to include descriptive summaries. - Improved overall documentation structure to facilitate better understanding of API functionalities. * removed repeating summary * Reordered tags and improved description relavancy * Refactor OpenAPI settings to enhance documentation clarity - Reorganized and updated descriptions for various API features, including "Cycles," "Modules," "States," "Users," "Work Items," and "Work Item Properties." - Removed redundant entries and improved the structure of feature descriptions to better reflect their functionalities and use cases. - Added new entries for "Work Item Types" and "Work Item Worklogs" to provide comprehensive coverage of work item management capabilities. * Enhance OpenAPI documentation for cycle and issue endpoints - Added response definitions for the `get_cycle_issues` and `delete_cycle_issue` methods in the CycleIssueAPIEndpoint to clarify expected outcomes. - Included additional response codes for the IssueSearchEndpoint to handle various error scenarios, improving the overall API documentation and usability. * Enhance serializer documentation across multiple files - Updated docstrings for various serializers including UserAssetUploadSerializer, AssetUpdateSerializer, and others to provide clearer descriptions of their functionality and usage. - Improved consistency in formatting and language across serializer classes to enhance readability and maintainability. - Added detailed explanations for new serializers related to project, module, and cycle management, ensuring comprehensive documentation for developers. * chore: remove duplicate summary * Refactor API endpoints for cycles, intake, modules, projects, and states - Replaced existing API endpoint classes with more descriptive names such as CycleListCreateAPIEndpoint, CycleDetailAPIEndpoint, IntakeIssueListCreateAPIEndpoint, and others to enhance clarity. - Updated URL patterns to reflect the new endpoint names, ensuring consistency across the API. - Improved documentation and method summaries for better understanding of endpoint functionalities. - Enhanced query handling in the new endpoint classes to streamline data retrieval and improve performance. * Refactor issue and label API endpoints for clarity and functionality - Renamed existing API endpoint classes to more descriptive names such as IssueListCreateAPIEndpoint, IssueDetailAPIEndpoint, LabelListCreateAPIEndpoint, and LabelDetailAPIEndpoint to enhance clarity. - Updated URL patterns to reflect the new endpoint names, ensuring consistency across the API. - Improved method summaries and documentation for better understanding of endpoint functionalities. - Streamlined query handling in the new endpoint classes to enhance data retrieval and performance. * Refactor asset API endpoint methods and introduce new status enums - Updated the GenericAssetEndpoint to only allow POST requests for asset creation, removing the GET method. - Modified the get method to require asset_id, ensuring that asset retrieval is always tied to a specific asset. - Added new IntakeIssueStatus and ModuleStatus enums to improve clarity and management of asset and module states. - Enhanced OpenAPI settings to include these new enums for better documentation and usability. * enforce naming convention * Added LICENSE to openapi spec * Refactor issue type and property API endpoints for improved clarity and functionality - Renamed existing API endpoint classes to more descriptive names such as IssueTypeListCreateAPIEndpoint, IssueTypeDetailAPIEndpoint, IssuePropertyListCreateAPIEndpoint, and IssuePropertyDetailAPIEndpoint. - Updated URL patterns to reflect the new endpoint names, ensuring consistency across the API. - Enhanced method summaries and documentation for better understanding of endpoint functionalities. - Streamlined query handling in the new endpoint classes to improve data retrieval and performance. * Enhance OpenAPI documentation for various API endpoints - Updated API endpoints in asset, cycle, intake, issue, module, project, and state views to include OpenApiRequest and OpenApiExample for better request documentation. - Added example requests for creating and updating resources, improving clarity for API consumers. - Ensured consistent use of OpenApi utilities across all relevant endpoints to enhance overall API documentation quality. * Enhance OpenAPI documentation for various API endpoints - Added detailed descriptions to multiple API endpoints across asset, cycle, intake, issue, module, project, state, and user views to improve clarity for API consumers. - Ensured consistent documentation practices by including descriptions that outline the purpose and functionality of each endpoint. - This update aims to enhance the overall usability and understanding of the API documentation. * Enhance OpenAPI documentation for issue types and properties by adding descriptions to various endpoints, including listing, creating, retrieving, updating, and deleting operations. Also, remove an unused import in the issue view file. * Enhance OpenAPI documentation for issue property and value endpoints by adding request examples for creating and updating issue types, properties, options, and values. This improves clarity for API consumers on expected request formats. * Update OpenAPI examples and enhance project queryset logic - Changed example fields in OpenAPI documentation for issue comments from "content" to "comment_html" to reflect the correct structure. - Introduced a new `get_queryset` method in the ProjectDetailAPIEndpoint to filter projects based on user membership and workspace, while also annotating additional project-related data such as total members, cycles, and modules. - Updated permission checks to use the correct attribute name for project identifiers, ensuring accurate permission handling. * Update OpenAPI documentation for issue comments to reflect changes in request format, replacing 'content' with 'comment_html'. Additionally, implement a new queryset method in ProjectDetailAPIEndpoint to enhance project filtering and member role annotations, improving data retrieval efficiency. Minor adjustments made to permission checks for project identifiers. * Enhance OpenAPI documentation and add response examples - Updated multiple API endpoints across asset, cycle, intake, issue, module, project, state, and user views to include new OpenApiResponse examples for better clarity on expected outcomes. - Introduced new parameters for project and issue identifiers to improve request handling and documentation consistency. - Enhanced existing responses with detailed examples to aid API consumers in understanding the expected data structure and error handling. - This update aims to improve the overall usability and clarity of the API documentation. * IssueProperty Nullenum fix * make api.plane.so default url in openapi spec * refactor: update terminology from 'issues' to 'work items' across multiple API endpoints for consistency and clarity * use common timezones from pytz for choices * refactor: remove unused imports from API views --------- Co-authored-by: pablohashescobar <nikhilschacko@gmail.com> Co-authored-by: NarayanBavisetti <narayan3119@gmail.com> Co-authored-by: Surya Prashanth <prashantsurya002@gmail.com>	2025-07-25 00:25:22 +05:30
Nikhil	7fe6ec7d38	fix: xmlsec version (#3644)	2025-07-11 14:31:01 +05:30
sriramveeraghanta	a193bb9aa5	fix: upgrade lxml version	2025-07-04 17:56:59 +05:30
sriramveeraghanta	46a46b7ef6	fix: merge conflicts	2025-07-04 15:37:59 +05:30
sriram veeraghanta	fdbe4c2ca6	chore: rename server to api (#7342)	2025-07-04 15:32:21 +05:30