mirror of
https://github.com/makeplane/plane.git
synced 2026-07-14 06:25:58 +02:00
ee7e457a39
* Enhancement: Add OpenSearch batch update tasks and monitoring features - Introduced new Celery tasks for processing batched OpenSearch updates and logging queue metrics. - Updated the task registry in `celery.py` to include `process-batched-opensearch-updates` and `log-opensearch-update-queue-metrics`. - Created a new module for batched search update tasks with comprehensive logging and error handling. - Added a management command for monitoring OpenSearch batch update queues with various options for real-time and detailed statistics. - Enhanced the OpenSearch documents module with new entity definitions and improved field handling for better search capabilities. - Updated settings to utilize the new batched signal processor for OpenSearch operations. - Removed deprecated signal handler to streamline the codebase. * Refactor: Update logging and registry access in search-related tasks - Changed logger name from "plane.api" to "plane.worker" for better context. - Simplified model search relevance checks by utilizing the registry's public API. - Enhanced cascade processing logic to improve clarity and efficiency. - Minor adjustments in the monitoring command to streamline confirmation checks. * Fix: Improve timestamp parsing in search queue monitoring * Refactor search queue monitoring command to support sub-commands * Update force-drain command to accept models as an option * Implement chunked processing for Redis queue updates - Added max_batch_size parameter to retrieve_all_queued_updates to limit the number of items processed at once, preventing memory issues. - Updated cleanup_stale_queue_for_model to process items in chunks, improving efficiency and stability for large queues. - Introduced OPENSEARCH_UPDATE_BATCH_MAX_SIZE setting for configurable batch processing limits. * Refactor: Replace retrieve_all_queued_updates with get_queued_updates_chunks for improved memory efficiency - Updated batched_search_update_task to utilize get_queued_updates_chunks, allowing for chunked processing of queued updates. - Modified related imports and settings to reflect the new chunking approach. - Enhanced queue processing logic to log progress and handle errors more effectively during chunk processing. * Changed the return value for unknown health statuses to use WARNING instead of HTTP_INFO * Multiple enhancements for chunk processing and metric logging - Reduced the chunk size for processing updates from 15000 to 1000 to improve performance. - Simplified the logging of queue metrics by focusing on essential metrics and removing unnecessary health status checks. - Updated timestamp handling to use epoch seconds for consistency. - Enhanced the alerting mechanism to only flag large queues, streamlining monitoring efforts. * Updated readme