Files
plane/docker-compose.yml
guru_sainath 1064e4fcdc Feat: Jira, Linear integration and GitHub importer (#1400)
* fix: silo service initial setup

* feat: moved controllers inside runner

* feat: created core definations for apps and context

* feat: added interfaces required for apps

* feat: moved shared definations

* feat: added logger as an injection for apps

* feat: changes to controllers

* feat: added vm based app execution inside manager

* feat: added dummy implementations for core constructs

* feat: moved controllers to routes

* feat: created worker moduler for processing message queue requests

* feat: added task manager defination

* feat: added context and manager separations

* feat: created defaults for the defination implementation

* feat: added async task management inside apps

* feat: removed unnecessary arguments in function calls

* feat: added context builder for app instances

* feat: added runtime type safety with zod

* feat: added packaged definition structure for silo apps

* feat: added package based configuration for engine and definitions

* feat: added custom eslint rule for throws decorator

* feat: added custom eslint rule for try catch

* feat: removed custom eslint config

* feat: added authentication controller inside engine

* feat: added auth in silo engine

* feat: migrated jira ui module to plane

* chore: removed jira app

* chore: moved engine components into src_engine directory

* feat: added base template inside new src directory

* feat: migrated worker interface with mq and redis store inside new src

* feat: created jira migrator and jira importer types

* chore: migrated worker's helper inside the base directory

* feat: added logic for booting up root worker

* feat: added jira and jira auth service inside jira app

* feat: added all transformers inside jira package

* feat: added authorization types for jira

* feat: added jira authentication oauth class

* feat: added jira transformer and pull mechanisms inside jira app

* feat: addded batching logic inside jira migrator

* feat: embedded silo sdk inside jira migrator

* feat: added plane migrator inside engine

* feat: added plane migrator for push inside migration controller

* feat: added controller methods for migration

* feat: added credentials and job routes in controller

* feat: added linear importer

* feat: added linear pull function as importer

* feat: added transformation for linear data

* feat: added pull mechanism for linear

* feat: attached linear data importer with migration controller

* fix: removed hardcoded jira from cycle and module migrator

* feat: fixed build errors

* chore: addeed example env

* feat: added authentication routes for jira

* feat: added linear route controller

* fix: restructuring

* fix: sdk configs setup

* fix: merge conflicts

* fix: sdk setup

* chore: added jira and linear importers and separate packages

* feat: moved transformation parts to linear package

* feat: decoupled jira logical parts with worker

* fix: linear silo app to use linear package

* fix: build errors and dependency resolution with packages in silo

* fix: module build errors in silo

* fix: linear authorization flow

* feat: added logic for segregated workers

* feat: attached task manager with the base starter

* feat: added migrations, query and schema into db directory

* feat: added linear importer and jira importer app structure

* feat: added silo core package

* chore: migrated worker and main engine controller inside apps

* fix: made linear integration working

* silo: added cors

* feat: added base64 state changes with jira

* chore: updated silo env

* fix: jira token cookie

* feat: added issue attachments in linear job

* feat: added credentials controller

* feat: added github package inside silo packages

* feat: added resource fetching in jira api service

* feat: added credentials locking

* feat: created resources endpoint for jira

* feat: added endpoints for getting jira data

* fix: credentials not working

* chore: exported jira types

* chore: added jira states

* fix: jira project pagination issue

* chore: initiated silo folder in web

* feat: added github routes and services in silo app

* fix: build updates

* fix: updated plane sdk and updated jira importer

* chore: updated the importer layout

* chore: removed as any from table component

* chore: integrated importer dashboard

* fix: tsup fixes

* fix: removed unnecessary files

* fix: removing tsup for building silo packages

* fix: build related issues

* fix: build issues

* fix: eslint fixes

* fix: silo build errors

* fix: silo app build errors types

* fix: reverting the cloud branch

* fix: updated package json in silo service

* fix: branch build cloud updated

* fix: build errors in apps while using sdk due to ts-alias paths

* fix: branch build cloud workflow fixes

* fix: docker compose setup updates

* fix: docker compose build fixes

* fix: docker build fixes envs updated in example file

* chore: updates folder structure and handled job services

* chore: resolved build errors in silo chore

* fix: docker compose cloud added

* fix: build process docker compose

* chore: updated jira workflow

* chore: handled the job start and jira dashboard

* chore: updated constants, file naming convention

* chore: resolved merge conflicts

* chore: integrated linear and updated jobs query

* feat: added hostname changes

* conflict: updated jira config and added issue transformation count in dashboard

* conflict: updated job

* chore: updated Jira status

* chore: updated Jira status

* fix: batch processing

* fix: batch key release

* chore: updated workflow for building images

* fix: detached silo build

* chore: updates linear and resolved build errors

* fix: batch key missing

* fix: linear workflow

* chore: updated linear queries

* fix: docker compose fixed for running silo services

---------

Co-authored-by: sriram veeraghanta <veeraghanta.sriram@gmail.com>
Co-authored-by: Henit Chobisa <chobisa.henit@gmail.com>
2024-10-08 23:16:28 +05:30

253 lines
5.6 KiB
YAML

x-silo-env: &silo-env
environment:
- PORT=3000
- BATCH_SIZE=${BATCH_SIZE:-60}
- MQ_PREFETCH_COUNT=${MQ_PREFETCH_COUNT:-5}
- APP_BASE_URL=${APP_BASE_URL:-http://web:3000}
- SILO_API_BASE_URL=${SILO_API_BASE_URL:-http://localhost:5050}
- DB_URL=${DB_URL:-postgresql://plane:plane@plane-db/silo}
- AMQP_URL=${AMQP_URL:-amqp://guest:guest@plane-mq:5672/}
- REDIS_URL=${REDIS_URL:-redis://plane-redis:6379/}
- SENTRY_DSN=${SENTRY_DSN:-" "}
- JIRA_CLIENT_ID=${JIRA_CLIENT_ID:-""}
- JIRA_CLIENT_SECRET=${JIRA_CLIENT_SECRET:-""}
- LINEAR_CLIENT_ID=${LINEAR_CLIENT_ID:-""}
- LINEAR_CLIENT_SECRET=${LINEAR_CLIENT_SECRET:-""}
services:
web:
container_name: web
build:
context: .
dockerfile: ./web/Dockerfile.web
args:
DOCKER_BUILDKIT: 1
restart: always
command: node web/server.js web
depends_on:
- api
admin:
container_name: admin
build:
context: .
dockerfile: ./admin/Dockerfile.admin
args:
DOCKER_BUILDKIT: 1
restart: always
command: node admin/server.js admin
depends_on:
- api
- web
space:
container_name: space
build:
context: .
dockerfile: ./space/Dockerfile.space
args:
DOCKER_BUILDKIT: 1
restart: always
command: node space/server.js space
depends_on:
- api
- web
live:
container_name: live
build:
context: .
dockerfile: ./live/Dockerfile.live
args:
DOCKER_BUILDKIT: 1
restart: always
command: node live/dist/server.js
silo:
<<: *silo-env
container_name: silo
build:
context: .
dockerfile: ./silo/Dockerfile.silo
args:
DOCKER_BUILDKIT: 1
restart: always
command: >
/bin/sh -c "
npm --prefix ./silo run db:migrate
node silo/start.js
"
depends_on:
plane-mq:
condition: service_healthy
# monitor:
# container_name: monitor
# build:
# context: .
# dockerfile: ./monitor/Dockerfile
# args:
# DOCKER_BUILDKIT: 1
# restart: always
api:
container_name: api
build:
context: ./apiserver
dockerfile: Dockerfile.api
args:
DOCKER_BUILDKIT: 1
restart: always
command: ./bin/docker-entrypoint-api-ee.sh
env_file:
- ./apiserver/.env
depends_on:
- plane-db
- plane-redis
mongodb:
image: "mongo"
restart: unless-stopped
volumes:
- "mongodb_data:/data/db"
environment:
- MONGO_INITDB_ROOT_USERNAME=${MONGO_INITDB_ROOT_USERNAME:-plane}
- MONGO_INITDB_ROOT_PASSWORD=${MONGO_INITDB_ROOT_PASSWORD:-plane}
worker:
container_name: bgworker
build:
context: ./apiserver
dockerfile: Dockerfile.api
args:
DOCKER_BUILDKIT: 1
restart: always
command: ./bin/docker-entrypoint-worker.sh
env_file:
- ./apiserver/.env
depends_on:
- api
- plane-db
- plane-redis
beat-worker:
container_name: beatworker
build:
context: ./apiserver
dockerfile: Dockerfile.api
args:
DOCKER_BUILDKIT: 1
restart: always
command: ./bin/docker-entrypoint-beat.sh
env_file:
- ./apiserver/.env
depends_on:
- api
- plane-db
- plane-redis
migrator:
container_name: plane-migrator
build:
context: ./apiserver
dockerfile: Dockerfile.api
args:
DOCKER_BUILDKIT: 1
restart: no
command: ./bin/docker-entrypoint-migrator.sh
env_file:
- ./apiserver/.env
depends_on:
- plane-db
- plane-redis
plane-db:
container_name: plane-db
image: postgres:15.7-alpine
restart: always
command: >
bash -c "
docker-entrypoint.sh postgres &
sleep 10
psql -U $$POSTGRES_USER -d $$POSTGRES_DB -c 'CREATE DATABASE silo;'
wait
"
volumes:
- pgdata:/var/lib/postgresql/data
env_file:
- .env
environment:
POSTGRES_USER: ${POSTGRES_USER}
POSTGRES_DB: ${POSTGRES_DB}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
PGDATA: /var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${POSTGRES_USER} -d ${POSTGRES_DB}"]
interval: 10s
timeout: 5s
retries: 5
start_period: 10s
plane-redis:
container_name: plane-redis
image: valkey/valkey:7.2.5-alpine
restart: always
volumes:
- redisdata:/data
plane-mq:
container_name: plane-mq
image: rabbitmq:management
restart: always
ports:
- "15672:15672"
env_file:
- .env
environment:
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER}
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD}
RABBITMQ_DEFAULT_VHOST: ${RABBITMQ_VHOST}
healthcheck:
test: rabbitmq-diagnostics -q ping
interval: 10s
timeout: 10s
retries: 5
volumes:
- rabbitmq_data:/var/lib/rabbitmq
plane-minio:
container_name: plane-minio
image: minio/minio
restart: always
command: server /export --console-address ":9090"
volumes:
- uploads:/export
environment:
MINIO_ROOT_USER: ${AWS_ACCESS_KEY_ID}
MINIO_ROOT_PASSWORD: ${AWS_SECRET_ACCESS_KEY}
# Comment this if you already have a reverse proxy running
proxy:
container_name: proxy
build:
context: ./nginx
dockerfile: Dockerfile
restart: always
ports:
- ${NGINX_PORT}:80
environment:
FILE_SIZE_LIMIT: ${FILE_SIZE_LIMIT:-5242880}
BUCKET_NAME: ${AWS_S3_BUCKET_NAME:-uploads}
depends_on:
- web
- api
- space
- admin
volumes:
pgdata:
redisdata:
uploads:
rabbitmq_data:
mongodb_data: