mirror of
https://github.com/colanode/colanode.git
synced 2026-02-24 03:49:48 +01:00
Add AI agent documentation (CLAUDE.md & AGENTS.md) (#295)
* add contributing md and coding agents md docs * remove contributing md from this branch * minor updates * resolve PR comments and add tests instructions
This commit is contained in:
Ylber Gashi
47
AGENTS.md
Normal file
47
AGENTS.md
Normal file
@@ -0,0 +1,47 @@
|
||||
# Repository Guidelines
|
||||
|
||||
## Architecture Overview
|
||||
Colanode is a local-first collaboration workspace. Clients keep a local SQLite cache and sync to the server (Fastify + Postgres + Redis) in the background for offline-first behavior. Real-time editing uses CRDTs (Yjs) to merge concurrent changes. Shared packages provide core types, sync logic, and UI; see `README.md` for product and hosting context.
|
||||
|
||||
## Project Structure & Module Organization
|
||||
- `apps/`: client and server apps (`server`, `web`, `desktop`, `mobile`).
|
||||
- `packages/`: shared libraries (`core`, `client`, `ui`, `crdt`).
|
||||
- `scripts/`: asset and seed tooling (postinstall runs from here).
|
||||
- `hosting/`: Docker Compose and Kubernetes (Helm) deploy configs.
|
||||
- `assets/`: repository images used in docs.
|
||||
|
||||
## Development Guide (Quick Start)
|
||||
- Install dependencies: `npm install`.
|
||||
- Prefer running tasks in individual app/package directories; repo-level scripts run the entire monorepo.
|
||||
- Run apps directly:
|
||||
- `apps/server`: `cp .env.example .env && npm run dev`
|
||||
- `apps/web`: `npm run dev` (Vite on port 4000)
|
||||
- `apps/desktop`: `npm run dev`
|
||||
- Local dependencies: `docker compose -f hosting/docker/docker-compose.yaml up -d`.
|
||||
|
||||
## Coding Guidelines
|
||||
- Ground changes in the existing codebase. Start from the closest feature and mirror its folders, naming, and flow.
|
||||
- Keep shared behavior in `packages/`; keep `apps/` thin and focused on wiring and UI.
|
||||
- Server routes use Fastify plugins with Zod schemas from `@colanode/core`. Update schemas and error codes before handlers.
|
||||
- Client operations follow the query/mutation pattern: define typed `type: 'feature.action'` inputs/outputs in `packages/client/src/queries` or `packages/client/src/mutations`, then wire handlers in `packages/client/src/handlers`.
|
||||
- Use Kysely (`database`) for SQL access and limit raw SQL.
|
||||
- UI styling uses Tailwind utilities, shared styles in `packages/ui/src/styles`, and shadcn components in `packages/ui/src/components/ui`. Prefer shared components over one-off styling.
|
||||
- Use `@colanode/*` imports and follow ESLint import grouping; keep filenames consistent with nearby code.
|
||||
|
||||
## Server Configuration
|
||||
- Config file location: set via `CONFIG` environment variable (e.g., `CONFIG=/path/to/config.json`).
|
||||
- If `CONFIG` is not set, server uses schema defaults from `apps/server/src/lib/config/`.
|
||||
- Template: `apps/server/config.example.json`.
|
||||
- Reference resolution in JSON:
|
||||
- `env://VAR_NAME` - resolves to environment variable (required, fails if not set).
|
||||
- `file://path/to/file` - reads and inlines file contents (useful for certificates).
|
||||
- Direct values - plain strings/numbers/booleans for non-sensitive settings.
|
||||
- Only `postgres.url` is required (defaults to `env://POSTGRES_URL`); all other settings have schema defaults.
|
||||
- For production: copy `config.example.json`, update values, mount it, and set `CONFIG` + required env vars.
|
||||
- Storage config via `storage.provider.type`: `file` (default), `s3`, `gcs`, or `azure`.
|
||||
- See `apps/server/src/lib/config/` for full schemas and validation.
|
||||
|
||||
## Testing
|
||||
- Tests live in `apps/server` and `apps/web` and run with Vitest.
|
||||
- Run `npm run test` in the relevant app directory; `npm run test` at the repo root runs them via Turbo.
|
||||
- Validate changes manually where tests do not apply and note verification steps.
|
||||
434
CLAUDE.md
Normal file
434
CLAUDE.md
Normal file
@@ -0,0 +1,434 @@
|
||||
# CLAUDE.md
|
||||
|
||||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||
|
||||
## Project Overview
|
||||
|
||||
Colanode is an open-source, local-first collaboration platform supporting real-time chat, rich text editing, customizable databases, and file management. It uses a sophisticated CRDT-based architecture (powered by Yjs) to enable offline-first operation with automatic conflict resolution.
|
||||
|
||||
## Commands
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Install dependencies (also runs postinstall script to generate emoji/icon assets)
|
||||
npm install
|
||||
```
|
||||
|
||||
Prefer running dev/build/compile/format commands inside the specific app or package directory.
|
||||
|
||||
**Note:** Tests exist for `apps/server` and `apps/web` and are run with Vitest. Prefer running tests in the relevant app directory; `npm run test` at the repo root runs the same suites via Turbo.
|
||||
|
||||
### Individual App Development
|
||||
|
||||
**Server:**
|
||||
|
||||
```bash
|
||||
cd apps/server
|
||||
cp .env.example .env # Configure environment variables
|
||||
npm run dev # Start server with hot reload
|
||||
|
||||
# Start dependencies (Postgres, Redis, Mail server) via Docker Compose
|
||||
docker compose -f hosting/docker/docker-compose.yaml up -d
|
||||
|
||||
# Include MinIO (S3-compatible storage) for testing
|
||||
docker compose -f hosting/docker/docker-compose.yaml --profile s3 up -d
|
||||
```
|
||||
|
||||
**Web:**
|
||||
|
||||
```bash
|
||||
cd apps/web
|
||||
npm run dev
|
||||
```
|
||||
|
||||
**Desktop:**
|
||||
|
||||
```bash
|
||||
cd apps/desktop
|
||||
npm run dev
|
||||
```
|
||||
|
||||
### Utility Scripts
|
||||
|
||||
Located in `scripts/`:
|
||||
|
||||
```bash
|
||||
cd scripts
|
||||
|
||||
# Generate emoji assets (run from scripts directory)
|
||||
npm run generate:emojis
|
||||
|
||||
# Generate icon assets (run from scripts directory)
|
||||
npm run generate:icons
|
||||
|
||||
# Seed database with test data (run from scripts directory)
|
||||
npm run seed
|
||||
```
|
||||
|
||||
Note: Generated emoji and icon assets are git-ignored. These are regenerated on `npm install` via the postinstall hook.
|
||||
|
||||
## Architecture
|
||||
|
||||
### Monorepo Structure
|
||||
|
||||
This is a Turborepo monorepo with npm workspaces:
|
||||
|
||||
- **`packages/core`** - Shared types, validation schemas (Zod), business rules, and node type registry. Foundation for all other packages.
|
||||
- **`packages/crdt`** - CRDT implementation wrapping Yjs. Provides type-safe document updates and conflict-free merging.
|
||||
- **`packages/client`** - Client-side services, local SQLite database schema, and API communication layer. Handles mutations, synchronizers, and offline support.
|
||||
- **`packages/ui`** - React components using TailwindCSS, Radix UI primitives, and TipTap editor. Shared between web and desktop apps.
|
||||
- **`apps/server`** - Fastify-based API server with WebSocket support. Manages Postgres database, Redis events, and background jobs.
|
||||
- **`apps/web`** - Web application (Vite + React + TanStack Router).
|
||||
- **`apps/desktop`** - Electron desktop application.
|
||||
- **`apps/mobile`** - React Native mobile app (experimental, not production-ready).
|
||||
- **`scripts/`** - Utility scripts for generating emojis, icons, and seed data.
|
||||
|
||||
### Local-First Architecture
|
||||
|
||||
**Core Principle:** All data operations happen locally first, then sync to the server in the background.
|
||||
|
||||
**Client Write Path:**
|
||||
|
||||
1. User makes a change (e.g., edits a document)
|
||||
2. Change is immediately applied to local SQLite database
|
||||
3. CRDT update is generated using Yjs (as binary `Uint8Array`)
|
||||
4. Update is stored in `mutations` table as a pending operation
|
||||
5. `MutationService` batches and sends mutations to server via HTTP POST
|
||||
6. Server validates, applies updates, and stores in Postgres
|
||||
7. Server broadcasts changes to other clients via WebSocket
|
||||
|
||||
**Client Read Path:**
|
||||
|
||||
- All reads happen from local SQLite (instant response)
|
||||
- `Synchronizer` services pull updates from server via WebSocket
|
||||
- Updates are applied to local database in background
|
||||
- UI reactively updates when local data changes
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `packages/client/src/services/workspaces/mutation-service.ts` - Mutation batching/syncing
|
||||
- `packages/client/src/services/workspaces/synchronizer.ts` - Real-time sync via WebSocket
|
||||
- `packages/client/src/databases/workspace/schema.ts` - Local SQLite schema
|
||||
- `apps/server/src/api/client/routes/workspaces/mutations/mutations-sync.ts` - Server mutation endpoint
|
||||
- `apps/server/src/services/socket-connection.ts` - WebSocket connection handler
|
||||
|
||||
### CRDT Integration (Yjs)
|
||||
|
||||
**Purpose:** Enable conflict-free collaborative editing with automatic merge resolution.
|
||||
|
||||
**Implementation:**
|
||||
|
||||
- `packages/crdt/src/index.ts` contains the `YDoc` class wrapping Yjs documents
|
||||
- Each node (page, database record, etc.) has a corresponding Yjs document
|
||||
- Updates are encoded as binary blobs (Base64 for storage, Uint8Array for processing)
|
||||
- Server merges concurrent updates automatically using Yjs CRDT semantics
|
||||
|
||||
**Storage Strategy:**
|
||||
Client storage maintains multiple layers:
|
||||
|
||||
1. **Current State** - JSON representation of latest merged state (for querying/UI)
|
||||
2. **Merged CRDT State** - Binary Yjs state in `node_states` / `document_states`
|
||||
3. **Pending Updates** - Local, unsynced CRDT updates in `node_updates` / `document_updates` kept separately so they can be reverted; once synced they are merged into `*_states` and removed
|
||||
|
||||
Server storage keeps current JSON state (`nodes` / `documents`) plus CRDT update history (`node_updates` / `document_updates`); background jobs merge older updates.
|
||||
|
||||
**Tables:**
|
||||
|
||||
- `nodes` / `documents` - Current state as JSON/JSONB
|
||||
- `node_states` / `document_states` - Merged CRDT state (client-side)
|
||||
- `node_updates` / `document_updates` - Pending local CRDT updates (client-side) and CRDT update history (server-side)
|
||||
|
||||
**Background Merging:**
|
||||
Server jobs periodically merge old updates to reduce storage (`apps/server/src/jobs/node-updates-merge.ts`).
|
||||
|
||||
### Database Synchronization
|
||||
|
||||
**Local (SQLite) ↔ Server (Postgres):**
|
||||
|
||||
Cursor-based streaming synchronization:
|
||||
|
||||
- Each data stream (users, nodes, documents, collaborations, etc.) has a `Synchronizer`
|
||||
- Synchronizers track a cursor (last synced revision number)
|
||||
- Client requests updates via WebSocket: `synchronizer.input { cursor: 12345 }`
|
||||
- Server responds with batch: `synchronizer.output { updates: [...], cursor: 12350 }`
|
||||
- Client applies updates to local database and persists new cursor
|
||||
|
||||
**Synchronizer Types:**
|
||||
|
||||
- `users` - User list changes
|
||||
- `collaborations` - Access control updates
|
||||
- `node.updates` - Node CRDT updates (per workspace root)
|
||||
- `document.updates` - Document CRDT updates (per workspace root)
|
||||
- `node.reactions` - Emoji reactions
|
||||
- `node.interactions` - Read receipts and activity tracking
|
||||
|
||||
**Key Files:**
|
||||
|
||||
- `packages/client/src/services/workspaces/synchronizer.ts`
|
||||
- `apps/server/src/synchronizers/*` - Server-side data fetchers
|
||||
- `apps/server/src/lib/event-bus.ts` - Event system for triggering syncs
|
||||
|
||||
### Node Type Registry
|
||||
|
||||
**Location:** `packages/core/src/registry/nodes/`
|
||||
|
||||
Each node type (space, page, database, message, etc.) defines:
|
||||
|
||||
- **Attribute Schema** - Zod schema for node metadata
|
||||
- **Document Schema** (optional) - Zod schema for collaborative content
|
||||
- **Permission Checks** - `canCreate`, `canUpdate`, `canDelete`, `canRead`
|
||||
- **Text Extraction** - For search indexing
|
||||
- **Mention Extraction** - For @mentions and notifications
|
||||
|
||||
**Example Node Types:**
|
||||
|
||||
- `space` - Top-level container
|
||||
- `page` - Rich text document
|
||||
- `database` - Structured data with custom fields and views
|
||||
- `record` - Database row
|
||||
- `database_view` - Saved database view
|
||||
- `chat` - Chat container
|
||||
- `channel` - Chat channel
|
||||
- `message` - Chat message
|
||||
- `file` - File attachment
|
||||
- `folder` - Organizational container
|
||||
|
||||
**Important:** When adding a new node type, register it in the appropriate registry file and ensure both client and server import it.
|
||||
|
||||
### Configuration System
|
||||
|
||||
**Server Configuration:**
|
||||
|
||||
The server uses a JSON-based configuration system with smart reference resolution:
|
||||
|
||||
- **Config File Location**: Set via `CONFIG` environment variable (e.g., `CONFIG=/path/to/config.json`)
|
||||
- **Default Behavior**: If `CONFIG` is not set, server uses schema defaults from `apps/server/src/lib/config/`
|
||||
- **Example Config**: See `apps/server/config.example.json` for a complete template
|
||||
|
||||
**Reference Resolution:**
|
||||
|
||||
The config system supports special prefixes for dynamic value loading:
|
||||
|
||||
- `env://VAR_NAME` - Resolves to environment variable at runtime (required, fails if not set)
|
||||
- `file://path/to/file` - Reads and inlines file contents at runtime (useful for certificates/secrets)
|
||||
- Direct values - Plain strings/numbers/booleans in JSON
|
||||
|
||||
**Example:**
|
||||
```json
|
||||
{
|
||||
"postgres": {
|
||||
"url": "env://POSTGRES_URL",
|
||||
"ssl": {
|
||||
"ca": "file:///secrets/postgres-ca.pem"
|
||||
}
|
||||
},
|
||||
"storage": {
|
||||
"provider": {
|
||||
"type": "s3",
|
||||
"endpoint": "env://S3_ENDPOINT",
|
||||
"accessKey": "env://S3_ACCESS_KEY",
|
||||
"secretKey": "env://S3_SECRET_KEY"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Required Configuration:**
|
||||
- Only `postgres.url` is truly required (defaults to `env://POSTGRES_URL`)
|
||||
- All other settings have sensible defaults in the Zod schemas
|
||||
- Storage defaults to `file` type with `./data` directory
|
||||
- Redis defaults to `env://REDIS_URL` but has fallback behavior
|
||||
|
||||
**For Docker/Production:**
|
||||
1. Copy `apps/server/config.example.json` to your own config file
|
||||
2. Update values (use `env://` for secrets, direct values for non-sensitive settings)
|
||||
3. Mount your config file and set `CONFIG=/path/to/mounted/config.json`
|
||||
4. Set required environment variables referenced by `env://` pointers
|
||||
|
||||
**For Local Development:**
|
||||
- Use `.env` file in `apps/server/` directory
|
||||
- Server will use schema defaults if no config file is provided
|
||||
- See `apps/server/.env.example` for available environment variables
|
||||
|
||||
**Config Validation:**
|
||||
- All config is validated using Zod schemas at startup
|
||||
- Validation errors show clear messages about missing/invalid values
|
||||
- Server exits immediately if config is invalid
|
||||
|
||||
**Client Apps:**
|
||||
|
||||
- Use standard `.env` files for build-time configuration
|
||||
- Runtime configuration fetched from server
|
||||
|
||||
## Testing
|
||||
|
||||
**Automated Tests (Vitest):**
|
||||
|
||||
- `apps/server`: `npm run test`
|
||||
- `apps/web`: `npm run test`
|
||||
- Repo root: `npm run test` (runs app tests via Turbo)
|
||||
- Focus manual verification on areas without test coverage
|
||||
- Include clear verification steps in pull request descriptions
|
||||
|
||||
## Development Tips
|
||||
|
||||
### Working with CRDTs
|
||||
|
||||
When modifying node or document schemas:
|
||||
|
||||
1. Update Zod schema in `packages/core/src/registry/nodes/<type>.ts`
|
||||
2. The CRDT layer automatically handles schema validation via `YDoc.update()`
|
||||
3. Test with multiple clients to verify conflict resolution
|
||||
4. Remember: updates are append-only, deletions use tombstones
|
||||
|
||||
### Debugging Synchronization
|
||||
|
||||
**Client-side:**
|
||||
|
||||
- Check `mutations` table for pending operations
|
||||
- Check `cursors` table for sync position
|
||||
- Use browser DevTools WebSocket tab to inspect messages
|
||||
|
||||
**Server-side:**
|
||||
|
||||
- Logs are in JSON format (Pino logger)
|
||||
- Look for `synchronizer.input` / `synchronizer.output` messages
|
||||
- Check `node_updates` table for stored updates
|
||||
- Verify revision numbers are incrementing
|
||||
|
||||
### Database Migrations
|
||||
|
||||
**Server (Postgres):**
|
||||
|
||||
- Schema defined in `apps/server/src/data/schema.ts`
|
||||
- Migrations live in `apps/server/src/data/migrations` and run via the Kysely migrator in `apps/server/src/data/database.ts`
|
||||
- Add a migration for schema changes; avoid manual production edits. For local dev, dropping the DB is a last resort.
|
||||
|
||||
**Client (SQLite):**
|
||||
|
||||
- Schema versioning handled in `packages/client/src/databases/workspace/schema.ts`
|
||||
- Migration logic in `packages/client/src/databases/workspace/migrations.ts`
|
||||
- Migrations run automatically on app startup
|
||||
|
||||
### Adding a New Node Type
|
||||
|
||||
1. Create schema in `packages/core/src/registry/nodes/<type>.ts`
|
||||
2. Define attribute schema, document schema (if collaborative), and permissions
|
||||
3. Register in `packages/core/src/registry/nodes/index.ts`
|
||||
4. Update server-side node creation logic in `apps/server/src/lib/nodes.ts`
|
||||
5. Add client-side service in `packages/client/src/services/`
|
||||
6. Create UI components in `packages/ui/src/components/`
|
||||
|
||||
### Storage Backends
|
||||
|
||||
Server supports multiple storage backends for files:
|
||||
|
||||
- **File** (default) - Local filesystem storage in `./data` directory
|
||||
- **S3** - AWS S3 or compatible (MinIO, DigitalOcean Spaces, etc.)
|
||||
- **GCS** - Google Cloud Storage
|
||||
- **Azure** - Azure Blob Storage
|
||||
|
||||
**Configuration:**
|
||||
|
||||
Configure via `storage.provider.type` in your config file. Examples:
|
||||
|
||||
**Filesystem (default):**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"provider": {
|
||||
"type": "file",
|
||||
"directory": "./data"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**S3-compatible:**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"provider": {
|
||||
"type": "s3",
|
||||
"endpoint": "env://S3_ENDPOINT",
|
||||
"accessKey": "env://S3_ACCESS_KEY",
|
||||
"secretKey": "env://S3_SECRET_KEY",
|
||||
"bucket": "env://S3_BUCKET",
|
||||
"region": "env://S3_REGION",
|
||||
"forcePathStyle": false
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**GCS:**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"provider": {
|
||||
"type": "gcs",
|
||||
"bucket": "env://GCS_BUCKET",
|
||||
"projectId": "env://GCS_PROJECT_ID",
|
||||
"credentials": "file:///secrets/gcs-credentials.json"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Azure:**
|
||||
```json
|
||||
{
|
||||
"storage": {
|
||||
"provider": {
|
||||
"type": "azure",
|
||||
"account": "env://AZURE_STORAGE_ACCOUNT",
|
||||
"accountKey": "env://AZURE_STORAGE_ACCOUNT_KEY",
|
||||
"containerName": "env://AZURE_CONTAINER_NAME"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See `apps/server/src/lib/storage/` for implementations and `apps/server/src/lib/config/storage.ts` for full schema.
|
||||
|
||||
## Common Patterns
|
||||
|
||||
### Service Layer Pattern
|
||||
|
||||
Services encapsulate business logic and coordinate between database and API:
|
||||
|
||||
- `packages/client/src/services/` - Client-side services
|
||||
- `apps/server/src/services/` - Server-side services
|
||||
|
||||
### Repository Pattern
|
||||
|
||||
Database access abstracted through clear interfaces:
|
||||
|
||||
- `packages/client/src/databases/` - Client database layer
|
||||
- `apps/server/src/data/` - Server database layer
|
||||
|
||||
### Event-Driven Updates
|
||||
|
||||
- Client: Uses TanStack DB for reactive data fetching alongside TanStack Query
|
||||
- Server: Uses EventBus (Redis-backed) for cross-instance communication
|
||||
- Background jobs via BullMQ for asynchronous processing
|
||||
|
||||
### Optimistic Updates
|
||||
|
||||
All mutations are optimistic:
|
||||
|
||||
1. Update local state immediately
|
||||
2. Show UI change instantly
|
||||
3. Send mutation to server in background
|
||||
4. On failure (after 10 retries), show error and optionally revert
|
||||
5. CRDT ensures eventual consistency even if server order differs
|
||||
|
||||
## Important Considerations
|
||||
|
||||
- **Generated Assets:** Emoji and icon files are generated during `npm install`. Don't commit these to git.
|
||||
- **TypeScript Source Imports:** Packages use TypeScript source directly (not compiled outputs) during development for faster iteration.
|
||||
- **Local-First Mindset:** Always assume network can fail. Design features to work offline first.
|
||||
- **CRDT Limitations:** Not all data types use CRDTs (e.g., messages and files use simpler database tables).
|
||||
- **Mobile App:** The `apps/mobile` is experimental and not production-ready.
|
||||
- **Performance:** For large workspaces, synchronizers per root node can cause memory pressure. Monitor closely.
|
||||
Reference in New Issue
Block a user