* add types for clickup entities
* add api service for clickup
* add pull and transform methods
* clickup folder
* add migrator and bulk transformer for clickup
* add clickup apk service for store
* add clickup data store
* add UI components and page clickup importer
* add UI components and page clickup importer
* fix bugs related to clickup import labels attachment issue type
* fix duplicate comments
* resolve plane client conflict
* content parser for description
* add relation map update for import job
* fix issues regarding custom field types options and values
* add translations + feature flag for clickup
* fix missing integration type export
* fix relation mapping across projects
* fix false comment actors in data import task
* fix cancel import job translation
* fix cancel import job translation
* fix rate limit issues for clickup import
* push unique work items to transform job
* handle username null in members
* increase refetch timing for jobs
* add rate limit fixes for pull in import job
* remove protect in pull job as not required now
* clickup additional data migrator for comments and attachments
* clickup additional migrator setup
* comment clickup importer
* fix status mapping for custom status in list n folders
* fix issues with clickup additional data migrator
* add changes to etl pull method clickup
* increase the rate limit for oauth tokens
* add pulling percent logs
* add batching to pull in clickup additional migrator
* add mq changes for error handling from preview
* Add migration for completed batch count + fix issues with progress of clickup additional migrator
* add support for external id and source in project and state external apis
* creation of project and state before transform if missing
* UI changes to support only space selection and start all folder import job
* Bug fixes for enhanced ClickUp importer with multi folder import; Translations; UI changes to support enhanced clickup importer
* migration to support enhanced clickup changes
* add emoji for project + no priority fix
* add logs to debug job stuck issue
* batch api calls to apiserver
* Fix edge-case in data import task to handle empty batches, preventing jobs from getting stuck in "TRANSFORMING" state.
* add retry logic for job finished in race condn and last batch failure
* feat: Update API routes with count increment endpoint, refactor import report logic, and enhance logging for job processing
* refactor: Improve logging in data import task and adjust batch completion logic for better handling of empty lists + UI fixes
* refactor: Enhance logging in data import task and streamline batch completion handling for empty issue lists
* refactor: remove batch locks
* refactor: enhance error handling in data migration
* chore: remove storing of members in config, skip sending relation map in get jobs
* refactor: logs and labels transform
* fix no page batch completion + feature flag clickup
* fix duplicate state mapping and transforming stuck
* reduce batching in state creation
* fix: remove multer package and resolved build errors
---------
Co-authored-by: sriram veeraghanta <veeraghanta.sriram@gmail.com>
* [SILO-249] feat: notion importer phase 1 with zip manager and worker traversal (#3153)
* feat: added notion zip manager code
* feat: added endpoint for bulk create pages
* feat: added page service in sdk
* fix: temporary runner
* feat: added notion zip file node types
* feat: added file extractor and file tree creator methods
* feat: added zip file manager class for interacting with extractors
* feat: added key manager types
* feat: added notion data migrator class for processing nodes
* feat: added method to set map in the store
* feat: added initialization of s3 in worker base
* feat: registered notion worker in the manager
* feat: added env variables for s3
* fix: remove unused services
* fix: removed unnecessary package
* fix: moved options into types
* fix: cleanup and jsdoc comments
* fix: renamed types and enums for consistency
* fix: replaced console.log with logger
* fix: added ttl to caches
* fix: moved zip manager to be reusable
* feat: added distributed counter for keeping count of leaf nodes
* fix: broken down the notion importer into multiple phases
* fix arrow function for key
Co-authored-by: Surya Prashanth <prashantsurya002@gmail.com>
* fix: eslint issues
* fix: unused variables and expression call
* fix: resolved eslint warnings
---------
Co-authored-by: Surya Prashanth <prashantsurya002@gmail.com>
* fix: build fix
* [SILO-254] feat: notion importer phase two implementation (#3169)
* feat: created endpoint for bulk operation on pages
* feat: added phase two implementation of notion migrator
* feat: added expected and ignored files list
* feat: added method for getting a specific map field
* fix: duplicate import
* feat: added page api endpoint
* feat: restructured and exported parser from core
* feat: added notion content parser extensions
* feat: added content parser initialization
* feat: added cleanup function in base migrator
* feat: refactored phases to include content parsing and cleanups
* chore: added content parser config
* feat: added cleanup function in zip manager to remove toc
* feat: added bluebird as dependency
* chore: remove test file
* fix: formatted code
* fix: decoupled content parser from `etl/core` to avoid circular dependency (#3204)
* fix: decoupled parser from etl/core and moved to etl/parser
* fix: modified linear importer for using parser from etl/parser
* fix: updated notion importer content parser imports
* [SILO-265] feat: Notion Importer UI and controller for generating presigned URL (#3237)
* feat: added notion controller for handling credentials save and uploads
* feat: added new keys and cached job and credential data
* feat: added notion controller and s3 init script
* feat: added UI changes for notion importer
* feat: added translation file and feature flags
* feat: added model changes to job and modified endpoint for pages
* feat: added start time to job
* fix: silo build failing
* feat: added notion importer UI component for uploading
* feat: added new extension for processing bookmark links
* fix: coderabbit suggestions
* fix: build fixes
* fix: condition
* fix: added page bulk endpoint under hmac and apiclient
* fix: root page when there is only one directory
* fix: added root directory path and error handling
* fix: add html page for each directory if not present
* feat: added feature flags for notion importer
* feat: added translations for all languagaes
* feat: added translations
* fixed: translations for russian
* [SILO-302] feat: migrated notion zip upload endpoints to apiserver (#3327)
* feat: added endpoint for silo uploads for notion importer
* feat: refactored controller to add start import route
* fix: removed policy for ttl in s3
* feat: modified frontend for using new endpoints from apiserver for uploads
* chore: added permission to silo asset endpoint
* feat: added notion importer feature flag to asset endpoint
* fix: used existing assets v2 for updating the asset upload
* fix: removed unused endpoints from silo asset view
* fix: moved notion importer into new settings page
---------
Co-authored-by: Surya Prashanth <prashantsurya002@gmail.com>
* feat: implement a generic oauth module for all callbacks
- Strategy based pattern to implement OAuth callbacks for any
third party integration
- Implement strategy for Plane's internal OAuth
- Add endpoint for fetching app installations
* add missing fields for integration helper in github controller
* pass client id, secret to strategy during init
* revert conflict changes
* use getPlaneAPIClient for consistency
* [SILO-142] Setup base for linear docs importer (#2934)
* feat: added page external api endpoint for creating and getting pages
* feat: added pages inside migration task
* feat: added pull documents and transform document functions inside linear
* feat: added pages inside generated payload pushed to queue
* feat: added separate job as linear_docs migrator
* fix: formatting
* fix: data_import_task broken to separate functions
* fix: separated github content-parser with linear content-parser
* fix: remove not existing page logic in process page
* fix: data import task to bulk create pages
* [SILO-145] feat: adding extendable content-parser for silo, supporting linear docs importer 🤠 (#2966)
* feat: added pages inside migration task
* feat: added pull documents and transform document functions inside linear
* feat: added pages inside generated payload pushed to queue
* feat: added separate job as linear_docs migrator
* fix: formatting
* fix: data_import_task broken to separate functions
* fix: separated github content-parser with linear content-parser
* feat: added content parser runner class and external parser extensions
* feat: added plane extension for plane side of parsing
* feat: added base-file-extension helper
* feat: added linear specific extensions and parser getter
* feat: modified linear transform and api service
* fix: modified base et for handling content-parsers opts
* fix: removed extra functions from image proxy extension
* fix: renamed to file-upload-helper
* fix: comments for processorts
* fix: moved get issue function inside mention parser
* fix: logic optimization
* feat: added modules extension in content-parser
* fix: mentions parser to work with fallback options
* fix: update user map type for linear
* [SILO-154] Added Linear Docs as summary element and optimized summary endpoint (#3010)
* feat: modified issue count endpoint to data summary endpoint
* feat: modified store and UI to display summary fields
* chore: resolved comments
* [SILO-153] feat: dispatch finished & linear docs step bound with existing with linear issue import step (#3009)
* feat: modified import task and controller to add a dispatch job
* feat: modified base importer worker in order to accomodate finished step
* feat: modified linear importer in order to handle custom completion step for job
* chore: resolved comments
* [SILO-134] RUNWAY Implement new OAuth flow for Github integration (#3089)
* oauth helper service + db helpers + modify plane sdk to accomodate oauth token
* fix indentation + create util for remove undefined in object
* add fields to workspace credential for oauth adapt
* get oauth token helper method
* auth callback flow changes to add plane oauth flow for github integration
* integrate github user and auth callback with oauth
* add user callback support with oauth for github integration
* remove token from cache on app disconnection
* Get Connection Details refactor change
* remove unwanted logs
* add missing translations
* replace plane app client id and secret fetching from env to db
* fix app get from db
* refactoring and comments resolved
* rename installation_id to plane_installation_id
* use db query methods and complete the app secret fetch flow
* remove unwanted logs and send missing values in update creds method
* move plane app db call just before oauth token call + models export in init
* Fix ssl reject error for pg
* add NODE_ENV option for ssl db config + management command to reset app_secrets + iteration fix
* fix oauth rate limit cache key breaking
* Fix redirect uri issue
* Fix pull request behaviour
* remove NODE_ENV check in DB connection
* feat: added external api to get a specific user
* feat: added user information in front of comment if not present
* chore: reverted changes for apis
* fix: user actor from webhook for getting name
* fix: remove unused import
* fix: added null check
* fix: change variable name
* [SILO-189] feat: added UI for connecting slack channels to plane projects (#3080)
* feat: added endpoint for fetching project updates connections
* feat: added service function for getting projects updates connection
* chore: added translations for slack project updates page
* feat: added type for response from slack conversation
* feat: added controller methods for getting project connections and channels
* feat: added service methods and project connections manager into store
* feat: added form component for project updates modal
* feat: added main list component for project updates and connected to root
* fix: error translation
* feat: added handler to address project updates
* chore: resolved comments
* feat: backend changes for slack project updates (#3105)
* fix: translations for project updates (#3107)
* feat: added types to optimize channel endpoint (#3108)
* fix: modified connection details for addressing user credentials
* fix: modified refresh credentials handler to upsert user credentials
* fix: modified types for both user responses
* fix: refactored comment handler to user issue details function
* chore: resolved comments
* chore: added logs
* fix: segregated type for token refresh
* fix: added null for issue details helper
* feat: added new blocks interface to ephemeral message posting
* feat: modified connection details helper in order to user credentials
* feat: modified controller to include userMap array
* feat: added account connection blocks
* feat: added missing credentials check and posting a message for connection
* feat: modified slack connection store for passing apiToken for state
* fix: resolved comments
* feat: added blocker for link work item command
* change pr behaviour method to use entityconnections for right state mapping
* renames and sanitize + modify test for PR automation state update
* remove console logs
---------
Co-authored-by: Surya Prashanth <prashantsurya002@gmail.com>
* oauth helper service
* oauth helper service + db helpers + modify plane sdk to accomodate oauth token
* fix indentation + create util for remove undefined in object
* remove await in helper methods
* workspace credential helper methods
* remove workspace_id from get workspace entity connection helper
* Get Connection Details refactor change
* fix indentation
* fix extra lines in end of file
* refactored to move integration related types in their specific files
* add application id copy button
---------
Co-authored-by: Saurabhkmr98 <saurabhkapur73@gmail.com>
Co-authored-by: Saurabh Kumar <70131915+Saurabhkmr98@users.noreply.github.com>
* fix: made redis instance singleton for being used multiple places
* fix: remove disconnection on shutdown
* fix: consumers throw error in start
* fix: remove extra methods and connection status
* fix: remove graceful shutdown
* fix: refactored pull-request handler into common behaviour
* fix: refactored gitlab integration to use common pr behaviour
* chore: added test to support the pull request behaviour
* chore: transformed to composition from inheritance
* fix: PR comments
* fix: extended type annotations and behaviour
