diff --git a/BREAKING_CHANGES.md b/BREAKING_CHANGES.md index 7c845eb..cad3081 100644 --- a/BREAKING_CHANGES.md +++ b/BREAKING_CHANGES.md @@ -1,6 +1,8 @@ # Breaking Changes -## `wrkflw run --event` requires change-set input by default (v0.7.3) +> The entries below ship in the next release (post-v0.7.3, currently unreleased on `main`). + +## `wrkflw run --event` requires change-set input by default (Unreleased) `wrkflw run` now supports trigger-aware filtering via `--event`, `--diff`, and `--changed-files`. When any of those flags is passed, the CLI runs a @@ -27,7 +29,7 @@ evaluation time. Users would then file "why didn't my workflow fire?" issues for the non-obvious reason that no change set had been supplied. Strict mode turns that silent failure into a loud, actionable error up front — it is the default countermeasure for the same class of silent -skip the rest of v0.7.3 patched iteratively. +skip the rest of the trigger-filter work patched iteratively. ### Impact @@ -75,10 +77,10 @@ Pick the option that matches your intent: ``` - **Legacy warn-and-proceed behavior (not recommended):** opt out with - `--no-strict-filter`. This restores the pre-v0.7.3 behavior of logging - a warning and running every workflow anyway. Use this only if your - scripts have already adapted to the old silent-skip semantics and you - cannot change them right now. + `--no-strict-filter`. This restores the pre-strict-filter behavior of + logging a warning and running every workflow anyway. Use this only if + your scripts have already adapted to the old silent-skip semantics and + you cannot change them right now. ### Prefilter exit codes @@ -99,7 +101,7 @@ Pick the option that matches your intent: --- -## Shell now matches GitHub Actions invocation (v0.7.3) +## Shell now matches GitHub Actions invocation (Unreleased) The `bash` shell now executes with `bash --noprofile --norc -e -o pipefail -c`, matching GitHub Actions behavior. The `sh` shell uses `sh -e -c`. This means: @@ -134,7 +136,7 @@ If a step intentionally tolerates command failures, either: --- -## EncryptedSecretStore serialization format (v0.7.3) +## EncryptedSecretStore serialization format (Unreleased) The `EncryptedSecretStore` struct in `crates/secrets/src/storage.rs` has changed its serialization format: diff --git a/README.md b/README.md index b3b352e..156d0b0 100644 --- a/README.md +++ b/README.md @@ -11,16 +11,20 @@ A command-line tool for validating and executing GitHub Actions workflows locall ## Features -- **TUI interface** — interactive terminal UI for browsing, running, and monitoring workflows +- **TUI interface** — interactive terminal UI with Workflows, Execution, DAG, Logs, Trigger, Secrets, and Help tabs - **Workflow validation** — syntax checks, structural validation, and composite action input cross-checking with CI/CD-friendly exit codes -- **Local execution** — run workflows using Docker, Podman, or emulation mode (no containers) -- **Job selection** — run individual jobs with `--job` flag or via TUI job selection mode +- **Local execution** — Docker, Podman, emulation, or sandboxed **secure emulation** (no containers) +- **Diff-aware filtering** — skip workflows whose `on:` block doesn't match the simulated event and changed file set +- **Watch mode** — rerun workflows automatically on file changes, with trigger-aware filtering +- **Job selection** — run individual jobs with `--job` or via TUI job selection mode - **Job dependency resolution** — automatic ordering based on `needs` with parallel execution of independent jobs -- **Action support** — Docker container actions, JavaScript actions, composite actions, and local actions -- **Reusable workflows** — execute caller jobs via `jobs..uses` (local or `owner/repo/path@ref`) +- **Expression evaluator** — evaluates `${{ ... }}` expressions including `toJSON`, `fromJSON`, `contains`, `startsWith`, etc. +- **Action support** — Docker container actions, JavaScript actions, composite actions (with output propagation), and local actions +- **Reusable workflows** — execute caller jobs via `jobs..uses` (local or `owner/repo/path@ref`) with output propagation +- **Artifacts, cache, and inter-job outputs** — `actions/upload-artifact`, `actions/download-artifact`, `actions/cache`, and `needs..outputs.*` - **GitHub context emulation** — environment variables, `GITHUB_OUTPUT`, `GITHUB_ENV`, `GITHUB_PATH`, `GITHUB_STEP_SUMMARY` - **Matrix builds** — full support for `include`, `exclude`, `max-parallel`, and `fail-fast` -- **Secrets management** — multiple providers (env, file, Vault, AWS, Azure, GCP) with masking and encryption +- **Secrets management** — multiple providers (env, file, Vault, AWS, Azure, GCP) with masking and AES-256-GCM encrypted storage - **Remote triggering** — trigger `workflow_dispatch` runs on GitHub or GitLab pipelines - **GitLab support** — validate and trigger GitLab CI pipelines @@ -49,6 +53,12 @@ wrkflw validate # Run a workflow wrkflw run .github/workflows/ci.yml + +# Rerun workflows automatically on file changes +wrkflw watch + +# List detected workflows and pipelines +wrkflw list ``` ## Usage @@ -87,6 +97,9 @@ wrkflw run --runtime podman .github/workflows/ci.yml # Run in emulation mode (no containers) wrkflw run --runtime emulation .github/workflows/ci.yml +# Run in sandboxed secure emulation +wrkflw run --runtime secure-emulation .github/workflows/ci.yml + # Run a specific job wrkflw run --job build .github/workflows/ci.yml @@ -97,6 +110,43 @@ wrkflw run --jobs .github/workflows/ci.yml wrkflw run --preserve-containers-on-failure .github/workflows/ci.yml ``` +### Trigger-aware execution + +Skip workflows whose `on:` block wouldn't fire for a given event/change set. Strict mode is on by default: `wrkflw run --event …` without `--diff` or `--changed-files` is rejected up front rather than silently skipping every `paths:`-gated workflow. + +```bash +# Auto-detect changed files from git (vs origin/HEAD, main/master, or HEAD~1) +wrkflw run --diff --event push .github/workflows/ci.yml + +# Pin the diff range +wrkflw run --diff --diff-base main --diff-head HEAD --event push .github/workflows/ci.yml + +# Supply changed files explicitly (e.g. from a CI wrapper) +wrkflw run --event push --changed-files src/main.rs,Cargo.toml .github/workflows/ci.yml + +# Simulate a pull_request — `--base-branch` is required under strict mode +wrkflw run --event pull_request --base-branch main --diff .github/workflows/ci.yml + +# Opt out of strict rejection (legacy warn-and-proceed) +wrkflw run --event push --no-strict-filter .github/workflows/ci.yml +``` + +See [BREAKING_CHANGES.md](BREAKING_CHANGES.md) for full migration notes. + +### Watch mode + +```bash +# Watch .github/workflows for changes and rerun affected workflows +wrkflw watch + +# Watch a specific path, simulate pull_request, and cap concurrency +wrkflw watch --event pull_request --base-branch main \ + --max-concurrency 2 --debounce 750 .github/workflows + +# Ignore extra directories on top of the built-in list +wrkflw watch --ignore-dir .terraform --ignore-dir coverage +``` + ### TUI ```bash @@ -107,20 +157,31 @@ wrkflw tui wrkflw tui --runtime podman ``` +**Tabs:** Workflows · Execution · DAG · Logs · Trigger · Secrets · Help. + **Controls:** | Key | Action | |-----|--------| -| `Tab` / `1-4` | Switch tabs (Workflows, Execution, Logs, Help) | -| `Up/Down` or `j/k` | Navigate | -| `Space` | Toggle selection | -| `Enter` | Run / View details | +| `Tab` / `Shift+Tab` | Switch tabs | +| `1`–`7` | Jump to tab by number | +| `w` / `x` / `l` / `h` | Jump to Workflows / Execution / Logs / Help | +| `↑`/`↓` or `k`/`j` | Navigate / scroll | +| `Space` | Toggle workflow selection | +| `Enter` | Run / view details | | `r` | Run selected workflows | -| `a` / `n` | Select all / Deselect all | -| `e` | Cycle runtime (Docker / Podman / Emulation) | +| `a` / `n` | Select all / deselect all | +| `Shift+R` | Reset workflow status | +| `Shift+J` | View jobs in workflow | +| `e` | Cycle runtime (Docker / Podman / Emulation / Secure Emulation) | | `v` | Toggle Execution / Validation mode | +| `d` / `D` | Toggle diff-aware filter / cycle simulated event | | `t` | Trigger remote workflow | -| `q` / `Esc` | Quit / Back | +| `,` | Open Tweaks overlay | +| `?` | Toggle help overlay | +| `q` / `Esc` | Quit / back | + +Logs tab adds `s` (search), `f` (filter), `c` (clear), `n` (next match). Trigger tab adds `p` (github↔gitlab), `b` (edit branch), `+` (add input), `c` (copy curl preview). ### Remote Triggering @@ -141,6 +202,7 @@ wrkflw trigger-gitlab --branch main --variable key=value | **Docker** (default) | Full container isolation, closest to GitHub runners | Production, CI/CD | | **Podman** | Rootless containers, no daemon required | Security-conscious environments | | **Emulation** | Runs directly on host, no containers needed | Quick local testing | +| **Secure Emulation** | Sandboxed host processes with filesystem/network restrictions | Running untrusted workflows without a container runtime | ## Reusable Workflows @@ -160,8 +222,9 @@ jobs: - Local refs resolve relative to the working directory - Remote refs are shallow-cloned at the specified `@ref` - `with:` entries become `INPUT_` env vars; `secrets:` become `SECRET_` +- Outputs from called jobs are merged back into `needs..outputs.*` -**Limitations:** outputs from called workflows are not propagated back; `secrets: inherit` is not supported; private repos for remote `uses:` are not yet supported. +**Limitations:** `secrets: inherit` is not supported; private repos for remote `uses:` are not yet supported; declared `on.workflow_call.outputs` is approximated by flattening all called-job outputs (the explicit mapping is not yet parsed). ## Secrets Management @@ -184,21 +247,24 @@ Supported providers: environment variables, file-based, HashiCorp Vault, AWS Sec - Workflow syntax validation with exit codes - Job dependency resolution and parallel execution - Matrix builds, environment variables, GitHub context -- Container, JavaScript, composite, and local actions -- Reusable workflows (caller jobs) +- `${{ ... }}` expression evaluation (`toJSON`, `fromJSON`, `contains`, `startsWith`, `success()`, `failure()`, etc.) +- Container, JavaScript, composite, and local actions (with composite-action output propagation) +- Reusable workflows (caller jobs) with output propagation into `needs..outputs.*` +- `actions/upload-artifact`, `actions/download-artifact`, and `actions/cache` (local-only, scoped to the run / workspace) - Environment files (`GITHUB_OUTPUT`, `GITHUB_ENV`, `GITHUB_PATH`, `GITHUB_STEP_SUMMARY`) +- Diff-aware trigger filtering (`--event`, `--diff`, `--changed-files`, `--base-branch`, `--activity-type`) +- Watch mode with trigger-aware re-execution - TUI and CLI interfaces - Container cleanup (even on Ctrl+C) ### Not Supported - GitHub encrypted secrets and fine-grained permissions -- `actions/cache` (no persistent cache between runs) -- Artifact upload/download between jobs -- Event triggers other than `workflow_dispatch` +- Event triggers other than `workflow_dispatch` for remote `trigger` command +- `secrets: inherit` on reusable workflow calls +- Private repos for remote `uses:` references - Windows and macOS runners - Job/step timeouts, concurrency, and cancellation - Service containers in emulation mode -- Reusable workflow output propagation (`needs..outputs.*`) ## Project Structure @@ -207,11 +273,13 @@ WRKFLW is organized as a Cargo workspace with focused crates: | Crate | Purpose | |-------|---------| | `wrkflw` | CLI binary and library entry point | -| `wrkflw-executor` | Workflow execution engine | +| `wrkflw-executor` | Workflow execution engine, expression evaluator, artifact/cache stores | | `wrkflw-parser` | Workflow file parsing and schema validation | | `wrkflw-evaluator` | Structural evaluation of workflow files | | `wrkflw-validators` | Validation rules for jobs, steps, triggers | | `wrkflw-runtime` | Container and emulation runtime abstractions | +| `wrkflw-trigger-filter` | `on:` block parsing and change-set matching | +| `wrkflw-watcher` | File watcher with trigger-aware re-execution | | `wrkflw-ui` | Terminal user interface | | `wrkflw-models` | Shared data structures | | `wrkflw-matrix` | Matrix expansion utilities | diff --git a/crates/README.md b/crates/README.md index 1f09497..e4e401b 100644 --- a/crates/README.md +++ b/crates/README.md @@ -7,15 +7,17 @@ This directory contains the Rust crates that make up the wrkflw workspace. | Crate | Purpose | |-------|---------| | **wrkflw** | CLI binary and library entry point | -| **executor** | Workflow execution engine (Docker, Podman, emulation) | +| **executor** | Workflow execution engine (Docker, Podman, emulation, secure emulation); `${{ }}` expression evaluator; artifact/cache/inter-job-output stores | | **parser** | Workflow file parsing and JSON Schema validation | | **evaluator** | Structural evaluation of workflow files | | **validators** | Validation rules for jobs, steps, triggers, matrix | | **runtime** | Container management and emulation runtime | +| **trigger-filter** | Parses `on:` blocks and matches them against simulated event context and changed-file sets | +| **watcher** | File watcher with trigger-aware re-execution for `wrkflw watch` | | **ui** | Terminal user interface (ratatui-based) | | **models** | Shared data structures (`ValidationResult`, GitLab models) | | **matrix** | Matrix expansion (`include`, `exclude`, `fail-fast`) | -| **secrets** | Secrets management with multiple providers and encryption | +| **secrets** | Secrets management with multiple providers and AES-256-GCM encryption | | **github** | GitHub API integration (list/trigger workflows) | | **gitlab** | GitLab API integration (trigger pipelines) | | **logging** | Thread-safe in-memory logging for TUI/CLI | diff --git a/crates/executor/README.md b/crates/executor/README.md index 9c170d5..0fa3847 100644 --- a/crates/executor/README.md +++ b/crates/executor/README.md @@ -1,13 +1,16 @@ ## wrkflw-executor -The execution engine that runs GitHub Actions workflows locally (Docker, Podman, or emulation). +The execution engine that runs GitHub Actions workflows locally (Docker, Podman, emulation, or secure emulation). - Job graph execution with `needs` ordering and parallel independent jobs -- Docker/Podman container steps and emulation mode +- Docker/Podman container steps, emulation, and sandboxed secure emulation - Run individual jobs via `target_job` / `--job` flag - GitHub Actions environment file support (`GITHUB_OUTPUT`, `GITHUB_ENV`, `GITHUB_PATH`, `GITHUB_STEP_SUMMARY`) with read-back -- Docker-based action resolution (container, JavaScript, composite, local) +- `${{ ... }}` expression evaluator (`toJSON`, `fromJSON`, `contains`, `startsWith`, `success()`, `failure()`, etc.) with GitHub / env / matrix / secrets / needs / steps context +- Action resolution for container, JavaScript, composite (with output propagation), and local actions - Job-level `container:` directive support +- Local `actions/upload-artifact`, `actions/download-artifact`, and `actions/cache` via shared artifact and cache stores +- Reusable workflow execution (`jobs..uses`, local or `owner/repo/path@ref`) with output aggregation into `needs..outputs.*` - **Used by**: `wrkflw` CLI and TUI ### API sketch @@ -16,15 +19,19 @@ The execution engine that runs GitHub Actions workflows locally (Docker, Podman, use wrkflw_executor::{execute_workflow, ExecutionConfig, RuntimeType}; let cfg = ExecutionConfig { - runtime: RuntimeType::Docker, + runtime_type: RuntimeType::Docker, verbose: true, preserve_containers_on_failure: false, + secrets_config: None, + show_action_messages: false, target_job: Some("build".to_string()), // run a single job }; let workflow_path = std::path::Path::new(".github/workflows/ci.yml"); let result = execute_workflow(workflow_path, cfg).await?; -println!("workflow status: {:?}", result.summary_status); +for job in &result.jobs { + println!("{}: {:?}", job.name, job.status); +} ``` Prefer using the `wrkflw` binary for a complete UX across validation, execution, and logs. diff --git a/crates/runtime/README.md b/crates/runtime/README.md index fa2f7e4..beba2d0 100644 --- a/crates/runtime/README.md +++ b/crates/runtime/README.md @@ -1,9 +1,13 @@ ## wrkflw-runtime -Runtime abstractions for executing steps in containers or emulation. +Runtime abstractions for executing steps in containers, on the host, or in a +local sandbox. -- Container management primitives used by the executor +- Container management primitives used by the executor (Docker, Podman) - Emulation mode helpers (run on host without containers) +- Secure emulation runtime: sandboxed host processes with filesystem and + network restrictions for running untrusted workflows without a container + runtime ### Example diff --git a/crates/trigger-filter/README.md b/crates/trigger-filter/README.md new file mode 100644 index 0000000..cce6e80 --- /dev/null +++ b/crates/trigger-filter/README.md @@ -0,0 +1,18 @@ +## wrkflw-trigger-filter + +Parses a GitHub Actions `on:` block and decides whether a workflow would fire +for a given event context and changed file set. Powers `wrkflw run --event …` +and `wrkflw watch`. + +- Parses `push`, `pull_request`, `pull_request_target`, `workflow_dispatch`, + `schedule`, and the other documented GHA events +- Matches `branches`, `branches-ignore`, `tags`, `tags-ignore`, `paths`, + `paths-ignore`, and `types:` filters +- Auto-detects the diff base (`origin/HEAD`, `main`, `master`, then `HEAD~1`) + when invoked without an explicit base ref +- Returns a structured reason on skip so the CLI/TUI can explain *why* a + workflow was filtered out + +Consumers: `wrkflw` CLI (`run`, `watch`) and `wrkflw-watcher`. Most users +should reach this functionality through the `--event` / `--diff` flags rather +than depending on the crate directly. diff --git a/crates/ui/README.md b/crates/ui/README.md index 7b2ccc7..dd858d5 100644 --- a/crates/ui/README.md +++ b/crates/ui/README.md @@ -2,9 +2,12 @@ Terminal user interface for browsing workflows, running them, and viewing logs. -- Tabs: Workflows, Execution, Logs, Help +- Tabs: Workflows, Execution, DAG, Logs, Trigger, Secrets, Help - Job selection mode: pick and run individual jobs within a workflow -- Hotkeys: `1-4`, `Tab`, `Enter`, `r`, `R`, `t`, `v`, `e`, `q`, etc. +- Diff-aware trigger filter (`d` / `D`) and Tweaks overlay (`,`) +- Log search (`s`), filter (`f`), and match navigation (`n`) +- Runtime cycling across Docker / Podman / Emulation / Secure Emulation (`e`) +- Hotkeys: `1`–`7` for tab jumps, `Tab`/`Shift+Tab`, `w/x/l/h` shortcuts, `Enter`, `r`, `Shift+R`, `Shift+J`, `t`, `v`, `?`, `q`, etc. - Optional: enabled via the `tui` cargo feature flag - Integrates with `wrkflw-executor` and `wrkflw-logging` @@ -17,7 +20,7 @@ use wrkflw_ui::run_wrkflw_tui; # tokio_test::block_on(async { let path = PathBuf::from(".github/workflows"); -run_wrkflw_tui(Some(&path), RuntimeType::Docker, true, false).await?; +run_wrkflw_tui(Some(&path), RuntimeType::Docker, /* verbose */ true, /* preserve_containers_on_failure */ false, /* show_action_messages */ false).await?; # Ok::<_, Box>(()) # })?; ``` diff --git a/crates/watcher/README.md b/crates/watcher/README.md new file mode 100644 index 0000000..e96c265 --- /dev/null +++ b/crates/watcher/README.md @@ -0,0 +1,17 @@ +## wrkflw-watcher + +File-system watcher with trigger-aware workflow execution. Backs the +`wrkflw watch` subcommand. + +- Debounced change detection via `notify` +- Per-workflow trigger cache (built on top of `wrkflw-trigger-filter`) so + only workflows whose `on:` block matches the change set are rerun +- Built-in ignore list (`.git`, `target`, `node_modules`, `.build`, `build`, + `dist`, `__pycache__`, `.tox`, `.mypy_cache`, `.pytest_cache`, `.venv`, + `venv`) plus user-supplied `--ignore-dir` values +- Concurrency cap per cycle, pending-event bound, and strict-filter mode + that rejects degraded event contexts with a loud error +- Graceful shutdown via the shared shutdown signal + +Consumers: `wrkflw` CLI (`watch` subcommand). Prefer the CLI unless you are +embedding the watcher into another tool. diff --git a/crates/wrkflw/README.md b/crates/wrkflw/README.md index 9950b1a..0e4b374 100644 --- a/crates/wrkflw/README.md +++ b/crates/wrkflw/README.md @@ -3,9 +3,11 @@ This crate provides the `wrkflw` command-line interface and a thin library surface that ties together all WRKFLW subcrates. It lets you validate and execute GitHub Actions workflows and GitLab CI pipelines locally, with a built-in TUI for an interactive experience. - **Validate**: Lints structure and common mistakes in workflow/pipeline files -- **Run**: Executes jobs locally using Docker, Podman, or emulation (no containers) +- **Run**: Executes jobs locally using Docker, Podman, emulation, or secure emulation, with optional diff-aware trigger filtering +- **Watch**: Reruns workflows on file changes with trigger-aware filtering - **TUI**: Interactive terminal UI for browsing workflows, running, and viewing logs - **Trigger**: Manually trigger remote runs on GitHub/GitLab +- **List**: Detect workflow and pipeline files in the repo ### Installation @@ -32,9 +34,17 @@ wrkflw validate path/to/flow-1.yml path/to/flow-2.yml path/to/workflows # Run a workflow (Docker by default) wrkflw run .github/workflows/ci.yml -# Use Podman or emulation instead of Docker +# Use Podman, emulation, or sandboxed secure emulation instead of Docker wrkflw run --runtime podman .github/workflows/ci.yml wrkflw run --runtime emulation .github/workflows/ci.yml +wrkflw run --runtime secure-emulation .github/workflows/ci.yml + +# Diff-aware filtering (skip workflows whose on: block doesn't match) +wrkflw run --diff --event push .github/workflows/ci.yml +wrkflw run --event pull_request --base-branch main --diff .github/workflows/ci.yml + +# Watch for changes and rerun affected workflows +wrkflw watch # Open the TUI explicitly wrkflw tui @@ -51,8 +61,12 @@ wrkflw tui --runtime podman - Flags: `--gitlab`, `--exit-code`, `--no-exit-code`, `--verbose` - **run**: Execute a workflow or pipeline locally - - Runtimes: `docker` (default), `podman`, `emulation` - - Flags: `--runtime`, `--job` (run a single job), `--jobs` (list jobs), `--preserve-containers-on-failure`, `--gitlab`, `--verbose` + - Runtimes: `docker` (default), `podman`, `emulation`, `secure-emulation` + - Flags: `--runtime`, `--job`, `--jobs`, `--preserve-containers-on-failure`, `--gitlab`, `--verbose` + - Trigger filter flags: `--event`, `--diff`, `--changed-files`, `--diff-base`, `--diff-head`, `--base-branch`, `--activity-type`, `--strict-filter` (default on), `--no-strict-filter` + +- **watch**: Watch a directory and rerun affected workflows on change + - Flags: `--runtime`, `--debounce`, `--event`, `--max-concurrency`, `--base-branch`, `--activity-type`, `--max-pending-events`, `--ignore-dir`, `--strict-filter` / `--no-strict-filter` - **tui**: Interactive terminal interface - Browse workflows, execute, and inspect logs and job details @@ -84,10 +98,14 @@ let cfg = ExecutionConfig { runtime_type: RuntimeType::Docker, verbose: true, preserve_containers_on_failure: false, + secrets_config: None, + show_action_messages: false, target_job: None, }; let result = execute_workflow(Path::new(".github/workflows/ci.yml"), cfg).await?; -println!("status: {:?}", result.summary_status); +for job in &result.jobs { + println!("{}: {:?}", job.name, job.status); +} # Ok::<_, Box>(()) # })?; ``` @@ -101,7 +119,7 @@ use wrkflw::ui::run_wrkflw_tui; # tokio_test::block_on(async { let path = PathBuf::from(".github/workflows"); -run_wrkflw_tui(Some(&path), RuntimeType::Docker, true, false).await?; +run_wrkflw_tui(Some(&path), RuntimeType::Docker, /* verbose */ true, /* preserve_containers_on_failure */ false, /* show_action_messages */ false).await?; # Ok::<_, Box>(()) # })?; ```