* feat: add recurring workitem models and update workitem template relationships
- Introduced `RecurringWorkitemConfiguration` and `RecurringWorkitem` models to manage recurring workitems.
- Updated `WorkitemTemplate` to include `parent_workitem_template` and `project_template` fields for better hierarchy and association.
- Enhanced model imports in `__init__.py` to include new recurring models.
* refactor: rename and enhance recurring workitem models
- Renamed `RecurringWorkitemConfiguration` to `RecurringWorkitemTask` and `RecurringWorkitem` to `RecurringWorkitemTaskLog` for clarity.
- Updated model fields to include `start_at`, `end_at`, `interval_seconds`, and `cron_expression` for improved scheduling flexibility.
- Added validation to ensure either `interval_seconds` or `cron_expression` is provided.
- Established a one-to-one relationship with Celery Beat's `PeriodicTask` for task management.
- Enhanced related names and help texts for better understanding and usability.
---------
Co-authored-by: Dheeraj Kumar Ketireddy <dheeru0198@gmail.com>
* Enhancement: Add OpenSearch batch update tasks and monitoring features
- Introduced new Celery tasks for processing batched OpenSearch updates and logging queue metrics.
- Updated the task registry in `celery.py` to include `process-batched-opensearch-updates` and `log-opensearch-update-queue-metrics`.
- Created a new module for batched search update tasks with comprehensive logging and error handling.
- Added a management command for monitoring OpenSearch batch update queues with various options for real-time and detailed statistics.
- Enhanced the OpenSearch documents module with new entity definitions and improved field handling for better search capabilities.
- Updated settings to utilize the new batched signal processor for OpenSearch operations.
- Removed deprecated signal handler to streamline the codebase.
* Refactor: Update logging and registry access in search-related tasks
- Changed logger name from "plane.api" to "plane.worker" for better context.
- Simplified model search relevance checks by utilizing the registry's public API.
- Enhanced cascade processing logic to improve clarity and efficiency.
- Minor adjustments in the monitoring command to streamline confirmation checks.
* Fix: Improve timestamp parsing in search queue monitoring
* Refactor search queue monitoring command to support sub-commands
* Update force-drain command to accept models as an option
* Implement chunked processing for Redis queue updates
- Added max_batch_size parameter to retrieve_all_queued_updates to limit the number of items processed at once, preventing memory issues.
- Updated cleanup_stale_queue_for_model to process items in chunks, improving efficiency and stability for large queues.
- Introduced OPENSEARCH_UPDATE_BATCH_MAX_SIZE setting for configurable batch processing limits.
* Refactor: Replace retrieve_all_queued_updates with get_queued_updates_chunks for improved memory efficiency
- Updated batched_search_update_task to utilize get_queued_updates_chunks, allowing for chunked processing of queued updates.
- Modified related imports and settings to reflect the new chunking approach.
- Enhanced queue processing logic to log progress and handle errors more effectively during chunk processing.
* Changed the return value for unknown health statuses to use WARNING instead of HTTP_INFO
* Multiple enhancements for chunk processing and metric logging
- Reduced the chunk size for processing updates from 15000 to 1000 to improve performance.
- Simplified the logging of queue metrics by focusing on essential metrics and removing unnecessary health status checks.
- Updated timestamp handling to use epoch seconds for consistency.
- Enhanced the alerting mechanism to only flag large queues, streamlining monitoring efforts.
* Updated readme
