# CLI (/docs/cli) OpenOpps exposes the `openopps` console script from `openopps.cli:app`. Treat the CLI as a local data pipeline: source catalogs become durable board records, board records become executable provider routes, provider routes become normalized jobs, and exports preserve the resulting ledger for audit or analysis. ## Invocation [#invocation] Run through the repository environment during development: ```bash uv sync just --list uv run openopps --help uv run openopps status ``` Install the current checkout as an editable uv tool when you want to call `openopps` directly: ```bash uv tool install -e . openopps --help openopps status ``` The editable tool uses the same runtime settings as `uv run`, including `OPENOPPS_DB_URL`, plugin allow lists, cache settings, and current working-directory-relative SQLite paths. ## Workflow [#workflow] The normal sequence is: 1. Run `sync` for source discovery, board route resolution, and job sync in order. 2. Use `sources sync`, `boards sync`, or `jobs sync` when rerunning one stage. 3. Inspect `providers coverage`, `providers audit`, or `admin providers registry` when route metadata is incomplete. 4. Use `jobs list`, `boards export`, or `jobs export` for analysis. Use `status` or `doctor` between steps to see counts, route readiness, cache state, plugin load state, and the next recommended action. ## Safety Classes [#safety-classes] | Class | Examples | What changes | | ----------------------------- | ------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------ | | Read-only local inspection | `status`, `doctor`, `sources list`, `providers coverage`, `plugins list`, `cache status` | Reads local SQLite/catalog state only. | | Writes local SQLite state | `examples seed`, `sources sync`, `boards sync`, `jobs sync`, `admin boards add`, `admin db init` | Creates or updates records under `OPENOPPS_DB_URL` and cache settings. | | Live upstream diagnostics | `admin sources test`, `providers health`, `admin providers probe-routes` | Calls public source or provider endpoints; dry-run unless the command documents `--apply`. | | Destructive local maintenance | `admin cache purge` | Deletes local cache records; scope with `--namespace` when possible. | `--apply` is intentionally absent from most everyday commands. When it appears on diagnostics such as route probing or health checks, treat it as the boundary between inspection and persistence. ## Command Groups [#command-groups] | Surface | Commands | Purpose | | ----------- | ----------------------------------------------- | -------------------------------------------------------------------------- | | `sync` | top-level | Run source discovery, board route resolution, and job sync in order. | | `status` | top-level | Report database, cache, plugin, route readiness, and next action. | | `doctor` | top-level | Same status payload with setup-oriented framing. | | `sources` | `list`, `show`, `sync` | Inspect aggregate discovery catalogs and import board records. | | `boards` | `sync`, `list`, `show`, `export` | Resolve, inspect, and export firm or company hiring board records. | | `jobs` | `sync`, `list`, `show`, `history`, `export` | Fetch, inspect, version, and export normalized public postings. | | `providers` | `health`, `coverage`, `audit` | Inspect live health, persisted coverage gaps, and adoption evidence. | | `cache` | `status` | Inspect the SQLite request cache. | | `plugins` | `list` | Inspect installed plugin entry points, capabilities, and failures. | | `examples` | `seed` | Seed deterministic synthetic demo data. | | `admin` | `sources`, `boards`, `providers`, `cache`, `db` | Advanced registration, route diagnostics, cache purge, and DB maintenance. | ## Option Conventions [#option-conventions] Examples use long flags for readability. The CLI also exposes script-friendly short aliases for high-traffic options: | Long flag | Aliases | Used by | | ----------------- | ---------- | --------------------------------------------------- | | `--source` | `-s`, `-S` | Source, board, job, provider, and audit scopes. | | `--board` | `-b`, `-B` | Job sync/list/export and route diagnostics. | | `--provider` | `-p`, `-P` | Board, job, provider, and route scopes. | | `--limit` | `-n`, `-N` | List and diagnostic result limits. | | `--json` | `-j`, `-J` | Machine-readable command output. | | `--output` | `-o`, `-O` | Export or no-DB output paths. | | `--format` | `-f`, `-F` | Export format selection. | | `--metrics-json` | `-m`, `-M` | Sync metrics output. | | `--refresh-cache` | `-r`, `-R` | Fresh upstream reads for cacheable request paths. | | `--verbose` | `-v`, `-V` | Detailed sync warnings instead of compact progress. | `--provider any` and `--provider all` both remove the provider filter. They are useful in reusable scripts that always pass a provider argument, but they do not mean “only providers named any/all.” ## Common Commands [#common-commands] ```bash uv run openopps status --json uv run openopps doctor --json uv run openopps sync a16z --metrics-json --refresh-cache uv run openopps sources sync a16z --metrics-json uv run openopps boards sync --source a16z --provider any --metrics-json uv run openopps jobs sync --provider ashbyhq --metrics-json --refresh-cache uv run openopps providers coverage --source a16z --provider any --json uv run openopps admin providers probe-routes --source a16z --provider any --limit 25 --json ``` Provider hints from source catalogs may lack the token or URL needed for job fetching. `admin providers probe-routes` tries candidate tokens from upstream slugs, remote ids, names, domains, and websites, then reports both matched routes and unknown boards. It is read-only unless `--apply` is passed. ## Board Filters [#board-filters] `boards list` and `boards export` share these filters: | Flag | Semantics | | ------------- | -------------------------------------------------------------------------- | | `--source` | Exact source key, such as `a16z` or `yc`. | | `--provider` | Exact detected board-provider route id. `any` and `all` remove the filter. | | `--market` | Case-insensitive substring match against board market tags. | | `--location` | Case-insensitive substring match against normalized board locations. | | `--domain` | Case-insensitive substring match against normalized board domains. | | `--has-jobs` | Keep boards with a source job hint, provider job hint, or synced job. | | `--min-staff` | Keep boards with `staff_count` greater than or equal to the value. | | `--max-staff` | Keep boards with `staff_count` less than or equal to the value. | | `--limit` | Apply a final limit after filters. | | `--json` | JSON output mode for `boards list`. | Use the exact persisted board key shown by `boards list`; provider requests are deduped before probing or job sync when overlapping source coverage points at the same provider route. ## Job Filters [#job-filters] `jobs list` and `jobs export` default to current active jobs with `--status open` across all boards and providers. Use `--status closed` or `--status all` for lifecycle audits. | Flag | Semantics | | ----------------------------- | ---------------------------------------------------------------------------------------------- | | `--source` | Exact source key via the job's board record. | | `--board` | Exact persisted board key from `boards list`. | | `--provider` | Exact job provider id. `any` and `all` remove the filter. | | `--location` | Case-insensitive substring match against normalized job locations. | | `--department` | Case-insensitive substring match against normalized department. | | `--team` | Case-insensitive substring match against normalized team. | | `--workplace-type` | Case-insensitive substring match against normalized workplace type. | | `--remote` | Case-insensitive exact match against normalized remote level: `Full`, `Hybrid`, or `None`. | | `--employment-type`, `--type` | Case-insensitive substring match against normalized employment type. | | `--salary-min` | Keep jobs whose normalized salary range overlaps this lower bound. | | `--salary-max` | Keep jobs whose normalized salary range overlaps this upper bound. | | `--skill` | Case-insensitive substring match against normalized skill name, level, or keywords. | | `--query` | Case-insensitive substring match across normalized title, company, and plain-text description. | | `--posted-after` | Inclusive `YYYY-MM-DD` lower bound for normalized `posted_at` dates. | | `--posted-before` | Inclusive `YYYY-MM-DD` upper bound for normalized `posted_at` dates. | | `--status` | Lifecycle filter: `open`, `closed`, or `all`. Defaults to `open`. | | `--limit` | Apply a final limit after filters. | | `--json` | JSON output mode for `jobs list`. | Date filters intentionally only match jobs whose normalized `posted_at` starts with `YYYY-MM-DD`, such as ISO timestamps. Relative provider text such as `Posted Yesterday` is not used for public filtering semantics. ## Exports [#exports] ```bash uv run openopps boards export --provider ashbyhq --has-jobs --format csv --output /tmp/openopps-boards.csv uv run openopps jobs export --source a16z --type full --format parquet --output /tmp/openopps-jobs.parquet ``` `boards export` and `jobs export` support: | Format | Use case | | --------- | ----------------------------------------------------------------------------------- | | `jsonl` | Streaming and audit-friendly line-delimited records. | | `csv` | Spreadsheet inspection and lightweight exchange. | | `parquet` | Analytics workflows with Polars, DuckDB, or warehouse ingestion. | | `sqlite` | Local relational handoff, reproducible filtered extracts, or direct SQLite clients. | JSONL exports stream line-delimited records and empty JSONL/CSV exports produce empty files. Empty Parquet exports produce a readable empty Parquet table. CSV exports neutralize spreadsheet formula-leading strings by prefixing a single quote. SQLite exports should keep the same flattened field contract as CSV/Parquet and store nested values as stable JSON strings. For schema, SQLite, and search-index details, see [Data Model](/docs/data-model). ## Troubleshooting Map [#troubleshooting-map] | Symptom | Command to run first | | -------------------------------------- | ------------------------------------------------------------------------------------------------- | | Local state looks empty | `uv run openopps status --json` | | A source returns no boards | `uv run openopps admin sources test --page-size 5 --refresh-cache` | | A board has provider hints but no jobs | `uv run openopps admin providers registry --include-missing --json` | | Route metadata is missing | `uv run openopps admin providers probe-routes --source --provider any --limit 25 --json` | | Cached data looks stale | Rerun the command with `--refresh-cache` or purge a specific namespace. | | Export output is empty | Check `status`, then rerun the matching `list` command with the same filters and `--json`. | # Configuration (/docs/configuration) OpenOpps loads settings from `OPENOPPS_` environment variables and an optional local `.env` file. Unknown `.env` keys are ignored. Treat configuration as local runtime wiring: the CLI writes to the SQLite URL you point it at and reads plugins from the Python environment that launched the command. ## Invocation Scope [#invocation-scope] Both invocation styles use the same settings model: ```bash uv run openopps status uv tool install -e . openopps status ``` The editable `uv tool install -e .` path makes the `openopps` command available directly from the current checkout. Relative SQLite URLs such as `sqlite:///openoppsdb.sqlite` are resolved by the running process, so keep your working directory and `OPENOPPS_DB_URL` explicit when switching between projects or smoke-test databases. ## Environment Variables [#environment-variables] | Variable | Default | Purpose | | ----------------------------------- | --------------------------------------- | ----------------------------------------------------------------------- | | `OPENOPPS_DB_URL` | `sqlite:///openoppsdb.sqlite` | Database URL used by storage commands and sync jobs. | | `OPENOPPS_MAX_CONNECTIONS` | `40` | Maximum HTTP connection pool size. | | `OPENOPPS_SOURCE_CONCURRENCY` | `4` | Source adapter concurrency for aggregate board discovery. | | `OPENOPPS_SOURCE_TIMEOUT_SECONDS` | `900.0` | Maximum seconds one source adapter can run before timed-out skip. | | `OPENOPPS_SOURCE_FRESHNESS_SECONDS` | `0.0` | Skip recently synced source catalogs during unscoped full-sync retries. | | `OPENOPPS_BOARD_CONCURRENCY` | `16` | Board-level concurrency during sync workflows. | | `OPENOPPS_PROVIDER_CONCURRENCY` | `12` | Provider job-fetch concurrency across job-capable routes. | | `OPENOPPS_WORKDAY_CONCURRENCY` | `2` | Conservative limit for public Workday CXS careers-site requests. | | `OPENOPPS_DB_BATCH_SIZE` | `500` | Batch size for SQLite writes. | | `OPENOPPS_HTTP_TIMEOUT` | `30.0` | HTTP timeout in seconds. | | `OPENOPPS_RETRY_ATTEMPTS` | `3` | Number of retry attempts for retriable upstream requests. | | `OPENOPPS_USER_AGENT` | `openopps/0.1 openopps@wyattowalsh.com` | User agent sent to public source and provider endpoints. | | `OPENOPPS_CACHE_ENABLED` | `true` | Enables the shared JSON request cache in the SQLite database. | | `OPENOPPS_CACHE_TTL_SECONDS` | `3600` | Default cache freshness window in seconds. | | `OPENOPPS_CACHE_REFRESH` | `false` | Bypasses cache reads while still updating successful responses. | | `OPENOPPS_CACHE_STALE_ON_ERROR` | `false` | Allows eligible stale cache data after retryable upstream errors. | | `OPENOPPS_PLUGIN_AUTOLOAD` | `false` | Execute every discovered plugin entry point without allow-listing. | | `OPENOPPS_PLUGIN_DISABLED` | empty | Comma-separated plugin entry-point names to skip. | | `OPENOPPS_PLUGIN_ALLOWED` | empty | Comma-separated plugin entry-point names allowed to execute. | | `OPENOPPS_NO_INTRO` | unset | Suppress the interactive startup portal animation. | ## Examples [#examples] ```bash OPENOPPS_DB_URL=sqlite:///data/openopps.db uv run openopps admin db status OPENOPPS_PROVIDER_CONCURRENCY=4 uv run openopps jobs sync --provider workday --profile OPENOPPS_USER_AGENT='openopps/0.1 (contact: jobs@example.com)' uv run openopps admin sources test yc ``` For repeated local use from the repository checkout, place values in `.env` at the repository root: ```dotenv OPENOPPS_DB_URL=sqlite:///openoppsdb.sqlite OPENOPPS_SOURCE_CONCURRENCY=4 OPENOPPS_SOURCE_TIMEOUT_SECONDS=900 OPENOPPS_SOURCE_FRESHNESS_SECONDS=0 OPENOPPS_PROVIDER_CONCURRENCY=12 OPENOPPS_CACHE_TTL_SECONDS=3600 OPENOPPS_PLUGIN_ALLOWED=trusted-plugin OPENOPPS_PLUGIN_DISABLED=broken-plugin ``` Unscoped `jobs sync` targets all persisted ready routes. Use explicit CLI filters such as `--source`, `--board`, or `--provider` for one-off scoped syncs. The docs [Jobs](/) and [Explorer](/explorer) surfaces read a committed static search snapshot generated from your local `kaggle/openoppsdb.sqlite` export. Refresh it with `just docs-search-index` after updating the database; see [Operations](/docs/operations) for validation commands. Docs-app telemetry is configured separately from CLI runtime settings. When instrumentation is enabled, keep collection env-gated: default to a no-op sink in local development, use `local-event-lake` for canonical raw events on persistent disk, and set `OPENOPPS_POSTHOG_PROJECT_API_KEY` only when a sanitized hosted product-analytics mirror is desired. Browser PostHog replay is a separate opt-in through `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY`; it remains gated by `NEXT_PUBLIC_OPENOPPS_TELEMETRY_ENABLED=true`, disables automatic PostHog pageview/autocapture events, masks text and inputs, disables network body/header capture, and leaves replay sampling plus trigger controls to the PostHog project. See [Data Model](/docs/data-model#telemetry-event-lake) for the event shape and [Operations](/docs/operations#telemetry-operations) for operating guidance. ## Flags vs Persistent Settings [#flags-vs-persistent-settings] Prefer flags for one run and environment variables for durable local policy: | Need | Prefer | | -------------------------------------------- | --------------------------------------- | | Refresh one upstream read | `--refresh-cache` | | Always bypass cache reads in a shell/session | `OPENOPPS_CACHE_REFRESH=true` | | Narrow one job sync | `--source`, `--board`, or `--provider` | | Lower Workday pressure across all runs | `OPENOPPS_WORKDAY_CONCURRENCY=1` or `2` | | Hide the animation once | `--no-intro` | | Hide the animation everywhere | `OPENOPPS_NO_INTRO=1` | By default, discovered plugins are visible but not executed. Set `OPENOPPS_PLUGIN_ALLOWED` to run specific trusted entry points. If `OPENOPPS_PLUGIN_AUTOLOAD=true`, every discovered plugin is eligible unless its entry-point name appears in `OPENOPPS_PLUGIN_DISABLED`. Disabled entries are still skipped even if also allow-listed. For CLI one-offs, prefer command flags such as `--source` and `--refresh-cache` over persistent environment changes. Do not commit `.env` files with private or environment-specific values. # Contributing (/docs/contributing) OpenOpps changes should keep the CLI, docs, generated artifacts, OpenSpec, and validation recipes aligned. The project is pre-release, but the local workflow should still be reproducible and reviewable. ## Local Setup [#local-setup] ```bash uv sync just --list uv run openopps --help cd docs && pnpm install ``` Use `uv run openopps ...` inside the repository checkout. Install the editable tool only when you want the `openopps` command available directly: ```bash uv tool install -e . openopps status ``` ## Validation [#validation] Use `just` from the repository root for local parity with GitHub Actions: ```bash just quick just ci just lock-check just openspec-validate-all just docs-check just docs-test just cli-help ``` The underlying commands remain direct and scriptable: ```bash uv run pytest uv run pytest --cov=openopps --cov-report=term-missing uv lock --check rtk npx -y @fission-ai/openspec@latest validate --all --strict cd docs && pnpm types:check cd docs && pnpm build cd docs && pnpm lint cd docs && pnpm test just docs-search-index-check ``` `just ci` includes lock-file checks, docs tests, docs lint, OpenSpec validation, coverage, Kaggle metadata generation, CLI help smoke checks, and formatting checks. `just docs-rtk-lint` is the explicit optional maintainer lint for `rtk`; it is not part of the default CI recipe. ## Docs Workflow [#docs-workflow] Docs content lives in `docs/content/docs/*.mdx`, and navigation order lives in `docs/content/docs/meta.json`. ```bash cd docs pnpm data:generate pnpm types:check pnpm build pnpm lint pnpm test ``` `pnpm data:generate` refreshes package-derived source/provider/export metadata. `pnpm types:check` also regenerates that metadata before Fumadocs MDX artifacts, Next.js route types, and TypeScript checks. Use `just docs-build` from the repository root for production docs build assurance; it also runs the API function trace check. The static jobs/explorer index is separate because it requires the ignored local SQLite snapshot: ```bash just docs-search-index just docs-search-index-check ``` Run the search-index check before release when refreshing `docs/public/data/openopps-search/`. ## Public Workflow Changes [#public-workflow-changes] Use OpenSpec for non-trivial changes to public workflows, generated asset formats, downstream agent tooling, docs generation, or validation behavior: ```bash rtk npx -y @fission-ai/openspec@latest list --json rtk npx -y @fission-ai/openspec@latest status --change prepare-v0-1-release --json rtk npx -y @fission-ai/openspec@latest validate --all --strict ``` When commands, workflows, or generated surfaces change, update the relevant MDX page, root README, nested `AGENTS.md`, `Justfile`, CI workflow, and OpenSpec change in the same logical workstream. ## Source and Provider Changes [#source-and-provider-changes] Source adapters discover candidate company boards. Provider adapters detect or fetch postings from public job-board providers. * Keep source adapters low-side-effect and explicit about upstream access. * Preserve source provenance in durable board records. * Keep route probing dry-run-first; persist with `--apply` only after matched routes are reviewed. * Add semantic tests for provider support and normalized output. * Use `providers coverage`, `providers audit`, and `admin sources yield` to evaluate persisted coverage before changing public claims. Installed Python plugins are not sandboxed and run in the same process as OpenOpps. Use `OPENOPPS_PLUGIN_ALLOWED` for trusted plugin entry points and `OPENOPPS_PLUGIN_AUTOLOAD=true` only in controlled environments. ## Data and Telemetry Contributions [#data-and-telemetry-contributions] Export and static-index changes must keep the data contract clear: * Update [Data Model](/docs/data-model) when entities, export formats, search-index fields, facets, suggestions, or telemetry events change. * Prefer generated counts and generated manifests over copied prose counts. * Keep SQLite, CSV, Parquet, and JSONL export semantics aligned. * Keep telemetry first-party, env-gated, size-capped, and sanitized. * Treat the local event lake as the canonical telemetry sink; optional dashboards or hosted adapters are mirrors. ## Secret Hygiene [#secret-hygiene] Keep credentials local. `.env`, `.env.*`, `.envrc`, Kaggle `kaggle.json`, local package-registry credential files, `.netrc`, key bundles, and token or credential JSON files are ignored; `.env.example` remains the tracked non-secret template. Do not print credentials in logs, docs, CI output, generated artifacts, or screenshots. Live Kaggle publishing remains a maintainer-only local action and is intentionally outside CI. # Data Model (/docs/data-model) OpenOpps keeps the operational ledger in local SQLite, publishes portable extracts for analysis, and serves the docs jobs/explorer surfaces from committed static artifacts. Use this page when you need to reason about data boundaries, counts, exports, or telemetry. ## Core Entities [#core-entities] | Entity | Meaning | Primary producer | | ------------------------------------------- | ------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------- | | `sources` | Aggregate discovery catalogs such as `a16z`, `accel`, `yc`, public-company indices, or ecosystem landscapes. | Packaged source adapters and plugins. | | `boards` | Firm or company hiring boards discovered from sources. | `sources sync`, `boards sync`, and admin board commands. | | `board_providers` | Provider route metadata between a board and a public job adapter. | Provider detection, route probing, and board sync. | | `jobs` | Stable posting identity and lifecycle fields. | Successful provider-route job syncs. | | `job_versions` | Normalized current or historical public posting content. | Job sync content hashing. | | `job_payload_snapshots` | Distinct raw listing/detail payload snapshots for replay and audit. | Job sync payload hashing. | | `job_sync_runs` and `job_sync_observations` | Per-route sync evidence for new, unchanged, changed, reopened, and closed outcomes. | Provider-route job syncs. | Job storage separates durable posting identity from changing content. The `jobs` table tracks status, first/last seen timestamps, close time, and current hashes. `job_versions` stores normalized user-visible content with child tables for locations, skills, skill keywords, and responsibility or qualification bullets. Raw payload drift updates payload snapshots and observations without creating a new user-visible version when normalized content does not change. ## Ingest surplus taxonomy (S1–S4) [#ingest-surplus-taxonomy-s1s4] Board ingestions capture more than first-class normalized columns. OpenOpps classifies the extra signal into four surplus classes: | Class | Where it lives today | Examples | | --------------------------- | ---------------------------------------------------- | ----------------------------------------------------- | | **S1 Raw-buried** | `raw_listing`, `raw_detail`, `job_payload_snapshots` | Greenhouse `metadata`, `requisition_id`, office trees | | **S2 Fetchable-not-called** | Board detail APIs not requested during sync | Greenhouse `pay_input_ranges`, org `/offices` trees | | **S3 Derivable** | Computable from normalized text | `seniority`, expanded skill catalog, `daysOpen` | | **S4 Sync-evidence** | `job_sync_runs`, `job_sync_observations` | Velocity, churn, reopen, median days open | Promotion policy: move high-cardinality S1 keys into typed `JobRecord` fields or `version.extra_payload` (`posting_kind`, `seniority`, `provider_extras`); keep full raw payloads in SQLite/Kaggle only. The docs search index uses **tiered detail shards**—T2 full body for indexable SEO jobs, T1 metadata-only for other open jobs—and never commits `payloadSnapshots`. ## Count Provenance [#count-provenance] OpenOpps has two generated count families: | Count family | Source | Used by | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------ | | Package catalog counts | `scripts/generate_docs_data.py` writes `docs/lib/generated/openopps-data.json`. | Source catalog, provider registry, export format, docs summary cards. | | Snapshot/search counts | `scripts/generate_docs_search_index.py` reads `kaggle/openoppsdb.sqlite` and writes `docs/public/data/openopps-search/`. | Jobs workbench, `/explorer`, snapshot freshness, filters, row totals, and preview/detail shards. | Do not copy counts into prose unless the sentence explicitly names the snapshot date or package source. Prefer generated components and generated manifest values so the site does not drift when sources, providers, locations, departments, or exports change. ## Exports [#exports] `boards export` and `jobs export` provide filtered extracts of normalized records: ```bash uv run openopps boards export --provider ashbyhq --has-jobs --format jsonl --output /tmp/openopps-boards.jsonl uv run openopps jobs export --remote Full --skill Python --format csv --output /tmp/openopps-jobs.csv uv run openopps jobs export --source a16z --salary-min 150000 --format parquet --output /tmp/openopps-jobs.parquet uv run openopps jobs export --status all --format sqlite --output /tmp/openopps-jobs.sqlite ``` | Format | Use case | Notes | | --------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------- | | `jsonl` | Streaming audit logs and line-oriented processing. | Preserves normalized values as JSON records. | | `csv` | Spreadsheet review and lightweight exchange. | Formula-leading strings are neutralized before writing. | | `parquet` | Polars, DuckDB, and analytics workflows. | Empty exports should remain readable empty tables. | | `sqlite` | Relational handoff, filtered local extracts, and portable query files. | Entity tables should use the flattened CSV/Parquet contract; nested values should be stable JSON strings. | The full OpenOppsDB snapshot remains the durable SQLite ledger. The docs and Kaggle bundle flows use `kaggle/openoppsdb.sqlite` as the local ignored input for static search artifacts and public dataset generation. ## Static Jobs and Explorer Index [#static-jobs-and-explorer-index] The docs app does not call provider APIs at runtime. It reads committed artifacts under `docs/public/data/openopps-search/`: | Artifact | Purpose | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `manifest.json` | Snapshot version, generated time, counts, chunks, facets, and provenance. | | Job chunks | Searchable latest job rows for the jobs workbench. | | Detail shards | Larger posting detail payloads loaded only after a job is selected. T1 shards carry metadata only; T2 shards add a bounded plain-text body (≤4000 characters, HTML stripped). | | Board/provider chunks | Explorer coverage, route, and source/provider inspection. | ### Indexable job criteria [#indexable-job-criteria] A job receives a **T2** detail shard (and appears in `jobs-indexable-ids.json`) only when all of the following hold at index generation time: * `status` is open (or unset) * `title`, `company`, and a non-empty description (plain or HTML) are present * at least one of `postedAt`, `firstSeenAt`, or `versionCreatedAt` is set * a safe absolute `postingUrl` or `applyUrl` is present (no credentials, fragments-only, or non-http schemes) All other open jobs still get T1 metadata-only detail shards for preview and filtering. Refresh the index after updating `kaggle/openoppsdb.sqlite`: ```bash just docs-search-index just docs-search-index-check ``` `just docs-search-index-check` requires the ignored local SQLite snapshot, regenerates `docs/public/data/openopps-search/`, and fails if the committed search artifacts still differ. If the SQLite file is absent, use package-derived docs checks and record the missing snapshot as the blocker for search-index parity. ## Facets and Suggestions [#facets-and-suggestions] Jobs and explorer filters should be generated from the static search manifest, not hardcoded UI lists. The manifest is the right place for canonical values and counts for: * sources and source aliases * providers and support levels * locations * departments and teams * companies * skills and skill keywords * workplace and employment types * job status and salary currency **Seniority** (`seniorities` facet): populated from each job row's `seniority` column. Values come from `version.extra_payload.seniority` when promoted during ingest; otherwise the index generator derives a label from normalized title and `job_versions.experience` text using the same `derive_seniority()` rules as `openopps.models`. **Days open** (`daysOpen` column): computed at index generation from `snapshotAt` minus `firstSeenAt` (non-negative integer days). This is an S3 derivable field and is not stored in SQLite job rows. Fuzzy matching should rank normalized generated suggestions first and only fall back to typed substring matching when the user enters a value that is not in the generated index. ## Provider surplus promotion [#provider-surplus-promotion] Provider adapters preserve full upstream payloads on `raw_listing` and `raw_detail`. v0.1 promotion focuses on **S1 list-endpoint fields** already fetched during sync: | Provider | Promoted surplus (initial) | | ---------- | ------------------------------------------------------------------------------------------------------------------------- | | Greenhouse | `metadata`, `requisition_id`, `language`, department/office trees; `posting_kind=prospect` when `internal_job_id` is null | | Workable | Split `raw_listing` (list POST) vs `raw_detail` (per-job GET) | | Others | See `openspec/changes/ingest-data-surplus/provider-promotion-manifest.json` | Derived **S3** fields such as `seniority` are computed from title/experience text without extra HTTP. **S4** sync aggregates surface in the search manifest `dashboard.sync` block when `job_sync_runs` is present in the snapshot. ## Telemetry Event Lake [#telemetry-event-lake] The docs app telemetry source of truth should be a first-party event lake. The collector can run as a no-op locally, append raw events when persistent storage is configured, mirror sanitized events to PostHog when configured, and compact local events into Parquet for DuckDB analysis. Recommended event shape: | Field | Meaning | | -------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------- | | `event_id` | Client-generated unique id for de-duping. | | `schema_version` | Telemetry schema version, starting at `1`. | | `sent_at` and `received_at` | Client and collector timestamps. | | `event_name` | Allowlisted event name such as `jobs.filters_changed` or `explorer.lineage_loaded`. | | `anonymous_id`, `session_id`, `page_id` | First-party anonymous identifiers. | | `context.path`, `context.viewport`, `context.screen`, `context.timezone`, `context.language`, `context.languages`, `context.userAgent` | Browser context retained by the current route allowlist. Raw `search`, `url`, and `referrer` fields are dropped. | | `request.path`, `request.headers`, `request.ip` | Server-side request context after header allowlisting, query removal, and IP drop/hash/raw policy. | | `entity_context` | Job, source, provider, filter, or explorer entity ids when relevant. | | `properties` | Event-specific allowlisted metadata. | | `redaction_count`, `payload_truncated` | Evidence that sanitization and size limits were applied. | `page_engagement` events include both cumulative counters (`durationMs`, `visibleDurationMs`, `interactionCount`) and additive counters (`durationDeltaMs`, `visibleDurationDeltaMs`, `interactionDeltaCount`) with a `reason` and `sequence`. Use the additive counters for aggregate dwell-time stories so route changes, hidden-tab flushes, and unmounts do not double-count the same page interval. Runtime sink modes: | Sink | Use case | Cost posture | | ------------------ | --------------------------------------------------- | --------------------------------------------- | | `noop` | Local development or deployments without storage. | Free; records nothing. | | `local-event-lake` | Canonical first-party telemetry on persistent disk. | Free when running on existing infrastructure. | The current `/api/telemetry` route treats any unsupported sink value as `noop`. PostHog forwarding is controlled separately with `OPENOPPS_POSTHOG_PROJECT_API_KEY`, receives only allowlisted sanitized properties, and is bounded by `OPENOPPS_POSTHOG_TIMEOUT_MS`. Browser PostHog session replay is an opt-in mirror controlled by `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY`; replay masks text and inputs, disables network body/header capture, respects PostHog project sampling and trigger controls, and does not change the first-party event schema. Cloudflare, Umami, or BI dashboards can mirror sanitized summaries downstream, but the canonical raw sink remains `local-event-lake`. Sanitize obvious secret-like values, cap event size, and make raw IP retention an explicit opt-in. Hashing IPs with a deployment salt is the default compromise when approximate abuse/debugging signal is useful. # Start Here (/docs) OpenOpps v0.1 is a CLI-only Python app for discovering firm hiring boards from aggregate sources, detecting public provider routes, syncing normalized jobs, caching repeated HTTP JSON requests, and exporting an auditable local opportunity ledger. It intentionally ships without a hosted service, browser app, prompt UI, or TUI.
Browse the committed snapshot Use the{" "} Jobs {" "} surface for open-role search and posting previews, or{" "} Explorer {" "} for source, provider, route, and data-quality analysis.
Discover Read source catalogs and persist durable board records.
Resolve Promote provider hints into executable public job routes.
Export Write normalized boards and jobs as JSONL, CSV, Parquet, or SQLite.
## Domain [#domain] * `sources` are aggregate discovery catalogs such as `a16z`, `accel`, `lsvp`, `sequoia`, `bvp`, `greylock`, `kleinerperkins`, `southparkcommons`, `signalfire`, and `yc`. * `boards` are firm/company hiring boards discovered from sources. * `jobs` are normalized postings fetched from boards. * `providers` are adapters that detect or fetch provider-specific boards, such as Ashby, Greenhouse, Lever, Workday, Workable, Teamtailor, BambooHR, Rippling, and WP Job Manager. * `cache`, `plugins`, and `examples` cover operational cache inspection, installed Python plugin discovery, and deterministic demo data. ## Install and Run [#install-and-run] Use `uv run` while working inside the repository checkout: ```bash uv sync just --list uv run openopps --help uv run openopps status ``` If you want `openopps` available directly from this editable checkout, install the repo as a uv tool from the repository root: ```bash uv tool install -e . openopps --help openopps status ``` `uv tool install -e .` installs the console entry point from the current checkout. Runtime settings such as `OPENOPPS_DB_URL` still control where local SQLite state is read and written. ## Contributor Command Map [#contributor-command-map] The root `Justfile` is the quickest way to discover local validation without hiding the underlying toolchain: ```bash just --list just quick just ci just openspec-validate-all just docs-check just cli-help ``` `just ci` mirrors the GitHub Actions validation lanes: Python tests, coverage, OpenSpec validation, generated docs data, docs type-check/build/lint, Kaggle metadata generation, and whitespace checks. Use the raw `uv`, `pnpm`, and OpenSpec commands from the reference pages when a CI failure needs exact reproduction. ## Safe Local Smoke Path [#safe-local-smoke-path] These commands seed deterministic demo records and do not hit upstream source or provider services: ```bash uv run openopps examples seed --seed 42 --boards 4 --jobs-per-board 2 --json uv run openopps jobs list --source example --json uv run openopps providers coverage --json ``` ## Live Quickstart [#live-quickstart] The commands below read public upstream catalogs and provider endpoints. They write local SQLite state unless a command is explicitly marked no-DB or dry-run. ```bash uv run openopps sources list uv run openopps admin sources test a16z uv run openopps sources sync a16z --metrics-json --refresh-cache uv run openopps sources sync accel --metrics-json uv run openopps sources sync greylock --metrics-json uv run openopps boards list --source a16z --limit 10 uv run openopps admin boards enrich --source a16z --json uv run openopps providers health --source a16z --provider any --limit 25 --json uv run openopps admin providers probe-routes --source a16z --provider any --limit 25 --json uv run openopps jobs sync --provider ashbyhq --metrics-json --refresh-cache uv run openopps cache status --json uv run openopps plugins list --json ``` Unscoped list, export, and sync commands use the full known superset unless narrowed by `--source`, `--board`, or `--provider`. Provider filters accept `any` and `all` as aliases for removing the provider filter, which keeps reusable scripts explicit without narrowing to one provider. Overlapping source coverage uses the persisted board keys shown by `boards list`, and provider requests are deduped before route probing or job sync. The relevant metrics include `duplicateRoutesSkipped`. ## Provider Levels [#provider-levels] | Level | Meaning | | ------------- | -------------------------------------------------------- | | `detect` | OpenOpps can preserve provider metadata and route hints. | | `jobs` | OpenOpps can fetch public jobs for the provider. | | `unsupported` | The provider is known only as raw metadata. | Ashby, Greenhouse, Lever, public Workday CXS, Workable, Teamtailor, BambooHR, Rippling, and explicit WP Job Manager boards are job-capable in v0.1 through public no-auth routes. Manatal and Gem hints remain detect-only until stable public fetching is added. See [Providers](/docs/providers) for the generated provider registry and route diagnostics. Ashby postings marked `isListed: false` are direct-link-only and are excluded from normal job sync output. ## Documentation Map [#documentation-map] | Page | Use it for | | ------------------------------------ | ----------------------------------------------------------------------------------------------------- | | [CLI](/docs/cli) | Command groups, common flags, JSON output, filters, and exports. | | [Configuration](/docs/configuration) | `OPENOPPS_` environment variables, defaults, `.env` loading, and runtime policy. | | [Data Model](/docs/data-model) | Sources, boards, providers, jobs, exports, static indices, counts, and telemetry event-lake guidance. | | [Providers](/docs/providers) | Source catalogs, provider support levels, route diagnostics, and limitations. | | [Operations](/docs/operations) | Storage, cache, exports, validation, telemetry operations, Kaggle, and troubleshooting workflows. | | [Contributing](/docs/contributing) | Local setup, validation, docs generation, source-batch rules, and review expectations. | | [Jobs](/) | Filter and preview open roles from the static search index. | | [Explorer](/explorer) | Analyze source/provider coverage, route health, freshness, and data quality. | ## Public Docs Routes [#public-docs-routes] The canonical jobs workbench is `/`, the canonical analytics explorer is `/explorer`, and thin job detail pages live at `/jobs/:id` for direct posting previews. Legacy `/jobs` redirects to `/`, and legacy `/docs/explorer` redirects to `/explorer`; do not add broad `/jobs/:path` redirects because detail pages must remain addressable. ## Route Probing Summary [#route-probing-summary] Provider hints from source catalogs may not include the token or URL needed for job fetching. `admin providers probe-routes` tries candidate tokens from upstream slugs, remote ids, names, domains, and websites, then reports both matched routes and unknown boards with the candidate tokens it tried. Its JSON output includes per-provider selected/matched counts, unresolved reason counts, and duplicate route skips. ```bash uv run openopps admin providers probe-routes --source a16z --provider all --limit 25 --json ``` Probing is read-only unless `--apply` is passed. Use [CLI](/docs/cli) for command flags and [Operations](/docs/operations) for local-state runbooks. # Operations (/docs/operations) Use this page for local CLI operations and docs validation while working on OpenOpps v0.1. There is no hosted runtime to initialize; durable state lives in local SQLite files controlled by `OPENOPPS_` settings.
Local state Sync and seed commands write to the database selected by OPENOPPS\_DB\_URL.
Dry-run diagnostics Route probing and health checks report first and persist only with --apply.
Export boundary JSONL, CSV, Parquet, and SQLite exports preserve normalized records for audit or analysis.
## Command Modes [#command-modes] Use `uv run openopps ...` inside the checkout. After `uv tool install -e .` from the repo root, the same examples can be run as `openopps ...` directly. ## Public docs surfaces [#public-docs-surfaces] The docs app exposes three v1 URL surfaces plus exact legacy redirects for moved routes. | URL | Purpose | | ----------- | ----------------------------------------------------------------- | | `/` | Interactive jobs board (search, filters, inline preview) | | `/explorer` | Dataset explorer across jobs, boards, and providers | | `/jobs/:id` | Static job detail pages for shareable permalinks and SEO metadata | | `/?job=:id` | In-app deep link that opens a job preview on the board | Bare `/jobs` redirects to `/`, and `/docs/explorer` redirects to `/explorer`. Trailing-slash variants normalize through the same canonical destinations. Job detail routes and `/jobs/sitemap/*.xml` remain available under the `/jobs/` prefix; do not add broad `/jobs/:path` redirects. ## Storage [#storage] SQLite is the default DB-backed storage mode: ```bash uv run openopps admin db init uv run openopps admin db status uv run openopps sync a16z --metrics-json uv run openopps sources sync a16z --metrics-json uv run openopps boards sync --source a16z --provider any --metrics-json uv run openopps jobs sync --provider any --metrics-json ``` The default database path is controlled by `OPENOPPS_DB_URL` and resolves to `openoppsdb.sqlite` for `sqlite:///openoppsdb.sqlite`. All sync commands in this section write local SQLite records unless they explicitly use `--no-db` or are documented as dry-run diagnostics. `admin db init` applies the current v0.1 Alembic schema head for the durable app SQLite database. HTTP response cache rows live in that same SQLite database and are managed by `cache.py`, not by the Alembic schema history. If a pre-release local SQLite database was stamped before the v0.1 schema was finalized, OpenOpps reports the missing schema columns and asks you to reset that local database or point `OPENOPPS_DB_URL` at a new SQLite file. Job storage separates stable posting identity from changing content. The `jobs` table tracks lifecycle fields such as `status`, `first_seen_at`, `last_seen_at`, `closed_at`, and current hashes. `job_versions` stores one normalized content state per posting, with child tables for locations, skills, skill keywords, and responsibility or qualification bullets. `job_payload_snapshots` stores distinct raw listing/detail payloads so raw upstream JSON is never discarded. Every successful provider-route job sync writes a `job_sync_runs` row and `job_sync_observations` rows for `new`, `unchanged`, `changed`, `reopened`, and `closed` outcomes. Raw payload drift means provider JSON changed but normalized content stayed the same, such as an ignored metadata key, key order change, or provider-internal token change. Raw-only drift updates payload snapshots and observations but does not create a new `job_versions` row. Use `status` or `doctor` to inspect the full local runtime state: ```bash uv run openopps status uv run openopps doctor --json ``` ## Cache [#cache] The shared request cache stores successful JSON request payloads in the configured SQLite application database for source adapters and job providers. It records deterministic request identity, payload hash, selected headers, freshness timestamps, ETag/Last-Modified validators, and stale-on-error eligibility. ```bash uv run openopps cache status uv run openopps admin cache purge --namespace http-json --json uv run openopps sources sync a16z --refresh-cache --metrics-json uv run openopps jobs sync --provider any --refresh-cache --metrics-json ``` `--refresh-cache` bypasses cache reads for that operation and lets successful responses update cache state. `admin cache purge` deletes cache rows. Scope it with `--namespace` when you only need to invalidate one cache family. ## Examples [#examples] Seed deterministic synthetic records for local demos and smoke tests without upstream network access: ```bash uv run openopps examples seed --seed 42 --boards 4 --jobs-per-board 2 --json uv run openopps jobs list --source example --json uv run openopps providers coverage --json uv run openopps admin sources yield --json ``` For a fully isolated smoke test, point `OPENOPPS_DB_URL` at a temporary SQLite file before seeding. ## No-DB Source Sync [#no-db-source-sync] For one-off source extraction without persistence, use explicit JSONL output: ```bash uv run openopps sources sync a16z --no-db --output /tmp/a16z-boards.jsonl ``` Human sync commands use a brief dynamic progress display by default. The top-level `sync` command reports source, board, and job stages in order; individual `sources sync`, `boards sync`, and `jobs sync` commands report the stage they own. Add `--verbose` when you need detailed provider warnings on stderr. Machine-readable modes such as `--metrics-json` keep stdout parseable. ## Exports [#exports] ```bash uv run openopps boards export --format jsonl --output /tmp/openopps-boards.jsonl uv run openopps boards export --provider ashbyhq --has-jobs --format jsonl --output /tmp/ashby-boards.jsonl uv run openopps jobs export --remote Full --skill Python --format csv --output /tmp/openopps-jobs.csv uv run openopps jobs export --status all --format jsonl --output /tmp/openopps-all-jobs.jsonl uv run openopps jobs export --source a16z --salary-min 150000 --format parquet --output /tmp/openopps-jobs.parquet uv run openopps jobs export --status all --format sqlite --output /tmp/openopps-jobs.sqlite ``` Use `jsonl` for auditability, `csv` for spreadsheet inspection, `parquet` for analytics workflows, and `sqlite` for portable relational handoff. JSONL exports stream records as they are encoded. Empty JSONL and CSV exports produce empty files; empty Parquet exports produce a readable empty Parquet table. CSV exports neutralize spreadsheet formula-leading strings by prefixing a single quote; JSONL and Parquet preserve values as-is for machine processing. SQLite exports should preserve the flattened export contract while storing nested values as stable JSON strings. `boards export` accepts the same board filters as `boards list`, including `--source`, `--provider`, `--market`, `--location`, `--domain`, `--has-jobs`, `--min-staff`, `--max-staff`, and `--limit`. `jobs export` accepts the same job filters as `jobs list`, including `--source`, `--board`, `--provider`, `--location`, `--department`, `--team`, `--workplace-type`, `--remote`, `--employment-type`/`--type`, `--salary-min`, `--salary-max`, `--skill`, `--query`, `--posted-after`, `--posted-before`, `--status`, and `--limit`. These filters use normalized current-version job fields; raw upstream payloads remain exported for auditability but are not part of primary filtering semantics. Job list and export commands default to `--status open`; pass `--status all` when you intentionally need closed jobs in an audit extract. ## Route Probing [#route-probing] The stable everyday route resolution command is `boards sync`, which enriches boards and applies successful missing route probes: ```bash uv run openopps boards sync --source a16z --provider all --limit 25 --metrics-json ``` The admin route-probing command remains dry-run-first for diagnostics: ```bash uv run openopps admin providers probe-routes --source a16z --provider all --limit 25 --json ``` Use `--apply` only after inspecting matched routes. ## Source Yield [#source-yield] Use source-yield reporting after source syncs, route probing, and job syncs when you need to compare which source families are turning into active job routes: ```bash uv run openopps admin sources yield --json uv run openopps admin sources yield --source sec-company-tickers ``` The report is offline and reads persisted SQLite records only. It does not fetch sources, probe routes, or sync jobs. Treat `yieldScore` as a local snapshot metric rather than a global benchmark unless the database was refreshed from a representative source set. ## Validation [#validation] Use `just` from the repository root for the contributor validation graph: ```bash just quick just ci just lock-check just openspec-validate-all just docs-check just docs-test just kaggle-meta just kaggle-bundle-check kaggle/openoppsdb.sqlite ``` Run Python tests from the repository root: ```bash uv run pytest uv run pytest --cov=openopps --cov-report=term-missing uv lock --check ``` Run docs checks from the docs app: ```bash cd docs pnpm data:generate pnpm data:generate:search pnpm types:check pnpm build pnpm lint pnpm test ``` `pnpm data:generate` refreshes package-derived provider/source metadata. `pnpm types:check` also regenerates that metadata before generating Fumadocs MDX artifacts, generating Next.js route types, and running TypeScript without emitting files. `pnpm lint` runs the Next ESLint surface. `pnpm test` runs the docs Vitest suite. `just docs-build` runs the production docs build and the function trace guard that keeps committed search artifacts out of API route bundles. Use `just docs-rtk-lint` from the repository root for the optional maintainer `rtk lint` surface; it requires `rtk` locally and is not silently skipped inside `just ci`. `pnpm data:generate:search` refreshes the committed static search index used by [Jobs](/) and [Explorer](/explorer) from `kaggle/openoppsdb.sqlite`; run it after updating the local OpenOppsDB snapshot. From the repository root you can also use `just docs-search-index`. The generated `docs/public/data/openopps-search/` tree is intentionally committed so the docs host can serve search, filters, dashboard aggregates, and job preview/detail shards without a live backend. Preview descriptions are limited to the normalized description fields present in that snapshot; when those fields are empty, the jobs surface shows metadata and links back to the source posting. `just docs-search-index-check` requires the ignored local `kaggle/openoppsdb.sqlite` snapshot and fails clearly when that file is unavailable. Do not remove committed search-index compatibility for the current artifact version unless the v4 generator has been run from that snapshot and `docs/public/data/openopps-search/` has been refreshed. Run OpenSpec checks from the repository root: ```bash rtk npx -y @fission-ai/openspec@latest list --json rtk npx -y @fission-ai/openspec@latest status --change prepare-v0-1-release --json rtk npx -y @fission-ai/openspec@latest validate --all --strict ``` OpenSpec-backed changes should keep proposal, design, specs, tasks, docs, nested `AGENTS.md`, CI, and just recipes synchronized. Use `instructions --json` when handing an OpenSpec task graph to another agent. ## Dependency And Secret Hygiene [#dependency-and-secret-hygiene] `uv.lock` and `docs/pnpm-lock.yaml` are committed reproducibility artifacts. Local validation and CI both run `uv lock --check`, and docs installs use `pnpm install --frozen-lockfile`. Renovate is configured at the repo root for Python `pyproject.toml`/`uv.lock` and docs npm/pnpm dependency maintenance. Keep credentials local. `.env`, `.env.*`, `.envrc`, Kaggle `kaggle.json`, local package-registry credential files, `.netrc`, key bundles, and token or credential JSON files are ignored; `.env.example` remains the tracked non-secret template. Do not print secrets in logs, docs, CI output, or generated artifacts. ## Outbound fetch threat model [#outbound-fetch-threat-model] OpenOpps source and job providers fetch public HTTPS endpoints from the local CLI. `openopps.http.assert_public_fetch_url` is a defense-in-depth guard for that local threat model: * URLs must be public `https://` endpoints with a host. * DNS is resolved once before the request; any address that resolves to a non-global-routable IP is rejected. * Redirect hops are origin-checked, but each hop performs its own DNS lookup. This guard is **not** rebinding-proof. httpx resolves DNS again at connect time, so a hostile or compromised resolver could return a public address during the pre-check and a private/metadata address when the socket connects (classic DNS TOCTOU / rebinding). Redirect chains widen the window because every hop repeats the lookup. v0.1 intentionally stops at lightweight validation rather than connect-time IP pinning or a trusted resolver policy. Treat the check as reducing accidental SSRF against RFC1918/link-local targets, not as a network sandbox. If you need stronger outbound controls later, pin the vetted addresses into the client connect path or run ingestion behind an egress policy you control. ## Telemetry Operations [#telemetry-operations] The docs app telemetry source of truth should be a first-party event lake. The default local mode is no-op; production-like deployments should write append-only events to persistent storage, compact them into Parquet with a separate maintenance job, and query them with DuckDB or another local analytical engine. | Sink | Operating model | Notes | | ------------------ | ---------------------------------------------------------- | ------------------------------------------------------- | | `noop` | Collects nothing. | Default for local development or hosts without storage. | | `local-event-lake` | Appends sanitized NDJSON events under UTC date partitions. | Canonical free sink when persistent disk is available. | The current collector accepts only `noop` and `local-event-lake` as `OPENOPPS_TELEMETRY_SINK` values. Cloudflare, Umami, or BI dashboards should be treated as downstream mirrors fed from the event lake or from sanitized aggregates, not as the canonical raw telemetry store. PostHog can be enabled as a free hosted product-analytics mirror by setting `OPENOPPS_POSTHOG_PROJECT_API_KEY`. The route forwards only allowlisted, sanitized events after OpenOpps redaction and never forwards raw posting bodies, request headers, or raw IPs. Forwarding is bounded by `OPENOPPS_POSTHOG_TIMEOUT_MS` and runs after successful `local-event-lake` writes, so a slow hosted mirror cannot block the canonical sink. Browser session replay is a separate public-key opt-in through `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY`; the browser SDK masks all inputs and text, blocks elements marked `data-openopps-private`, `data-telemetry-private`, or `data-sensitive`, disables network body/header capture, leaves sampling plus URL/event/linked-flag controls to PostHog project configuration, and keeps PostHog automatic pageview/autocapture events off so OpenOpps remains the event schema source of truth. Use PostHog for dashboards and masked replay while keeping `local-event-lake` as the canonical raw sink when persistent storage is available. Telemetry operations should preserve the event lake, cap payload sizes, record redaction/truncation evidence, and avoid raw IP retention. Keep event names and property keys allowlisted so exploratory UI instrumentation does not accidentally persist secrets from query strings, stack traces, or copied provider payloads. Page-engagement stories should sum `visibleDurationDeltaMs` and `durationDeltaMs`; cumulative `visibleDurationMs` and `durationMs` are retained for last-known-state debugging. | Variable | Default | Production rule | | ---------------------------------------------- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `NEXT_PUBLIC_OPENOPPS_TELEMETRY_ENABLED` | disabled | Enables the browser client only when explicitly set. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_PROJECT_API_KEY` | unset | Optional browser PostHog key for masked session replay; ignored unless browser telemetry is enabled. | | `NEXT_PUBLIC_OPENOPPS_POSTHOG_HOST` | `https://us.i.posthog.com` | Optional browser PostHog ingestion host. | | `OPENOPPS_TELEMETRY_SINK` | `noop` | Use `local-event-lake` only when persistent storage is available. | | `OPENOPPS_TELEMETRY_DIR` | unset | Required for `local-event-lake`; events are partitioned by UTC date. | | `OPENOPPS_TELEMETRY_SALT` | test/dev fallback | Required before `local-event-lake` accepts writes; rotate through deployment config, not source. | | `OPENOPPS_TELEMETRY_IP_MODE` | `hash` | `raw` is rejected when `NODE_ENV=production`; use `hash` or `drop`. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_MAX` | `120` | Public POST limit per salted request fingerprint. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_WINDOW_MS` | `60000` | Rate-limit window in milliseconds. | | `OPENOPPS_TELEMETRY_RATE_LIMIT_MAX_BUCKETS` | `4096` | Maximum in-process rate-limit fingerprints retained before oldest buckets are evicted. | | `OPENOPPS_TELEMETRY_TRUSTED_PROXY` | `none` | Valid values are `none`, `cloudflare`, `vercel`, and `forwarded`. Client IP headers are ignored unless a mode is selected; `cloudflare` trusts `cf-connecting-ip`, `vercel` trusts `x-vercel-forwarded-for`, and `forwarded` trusts the first `x-forwarded-for` hop only when your ingress strips client-supplied values. `x-real-ip` is not trusted by this route. | | `OPENOPPS_POSTHOG_PROJECT_API_KEY` | unset | Optional PostHog project key for sanitized hosted product analytics. | | `OPENOPPS_POSTHOG_HOST` | `https://us.i.posthog.com` | Optional PostHog ingestion host. | | `OPENOPPS_POSTHOG_TIMEOUT_MS` | `1500` | Best-effort PostHog forwarding timeout, clamped to 100-10000 ms. | ## OpenOppsDB Kaggle [#openoppsdb-kaggle] The generated Kaggle bundle lives in `kaggle/`. It contains `dataset-metadata.json`, `kernel-metadata.json`, the connected `openoppsdb-manager.ipynb` notebook, public example notebooks under `kaggle/starter/` and `kaggle/examples/`, and generated SQLite/CSV/Parquet artifacts when a data DB is supplied. `dataset-metadata.json` is the Kaggle UI source of truth for public file descriptions and field-level descriptors for every CSV and Parquet export. `openoppsdb.sqlite` remains directly readable and carries `openopps_tables`/`openopps_columns` metadata for SQLite clients; if Kaggle does not expose nested SQLite table previews for a fresh upload, use the mirrored CSV/Parquet exports for Kaggle-rendered table previews and field metadata. Full snapshot runs write private `sync_metrics.json`, `status.json`, `coverage.json`, and `snapshot-quality.json` evidence while validating the bundle, then prune those files before publishing. The live dataset recipes stage a temporary upload directory containing only Kaggle dataset control files plus `openoppsdb.sqlite`, `exports/csv/*.csv`, and `exports/parquet/*.parquet`; notebooks are pushed as separate Kaggle kernels. Run the local non-live generation and validation path from the repository root: ```bash just kaggle-meta PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db kaggle/openoppsdb.sqlite just kaggle-bundle-check kaggle/openoppsdb.sqlite ``` For a fresh local SQLite bundle seed, initialize the DB and run top-level source discovery before bundle generation: ```bash OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps admin db init OPENOPPS_DB_URL="sqlite:///$PWD/.tmp/openoppsdb-operational.sqlite" uv run openopps sync --metrics-json --refresh-cache PYTHONPATH=scripts uv run python -m openopps_kaggle --data-db .tmp/openoppsdb-operational.sqlite ``` The manager notebook should be scheduled in Kaggle as one daily full snapshot, for example `0 6 * * *`. Each run installs OpenOpps from `git+https://github.com/wyattowalsh/openopps.git@main` by default, reads public snapshot input from `wyattowalsh/openoppsdb`, copies the verified `openopps_kaggle` runtime package from the private `wyattowalsh/openoppsdb-manager-runtime` input, copies the newest `/kaggle/input/**/openoppsdb.sqlite` ledger into `/kaggle/working/openoppsdb/openoppsdb.sqlite`, may restore large columns from prior Parquet exports when upgrading legacy thin snapshots, rehydrates the public SQLite snapshot into a fresh operational Alembic schema when needed, and runs bounded `openopps jobs sync --metrics-json --freshness-seconds --limit`. It writes private status and coverage evidence, then runs `python -m openopps_kaggle` to backfill derived tables, export parquet-first artifacts, regenerate metadata, prune private evidence, stage the public upload directory, and attempt best-effort live Kaggle file metadata repair. Run `just kaggle-live-file-metadata` from a browser-authenticated local maintainer environment after live publishes when the Kaggle DataBundle checklist or column-description score must be repaired authoritatively. Refresh the private runtime dataset with the current `openopps_kaggle` package before pushing the manager. The notebook seeds bounded Kaggle runtime defaults for source freshness, job-route freshness, route limits, concurrency, connection limits, timeouts, and retries while allowing `OPENOPPS_` environment overrides. The scheduled Kaggle notebook environment must expose Kaggle API credentials before the manager starts. Configure `KAGGLE_USERNAME` and `KAGGLE_KEY`, or `KAGGLE_API_TOKEN`, as Kaggle notebook secrets/environment variables; otherwise the manager fails fast before running the expensive sync. Live deployment remains a local maintainer action with Kaggle CLI credentials: ```bash just kaggle-dataset-create just kaggle-dataset-version "OpenOppsDB daily snapshot" just kaggle-runtime-generator-version "OpenOppsDB manager runtime generator" just kaggle-notebook-push just kaggle-example-notebooks-push just kaggle-example-notebooks-status just kaggle-example-notebooks-pull-check just kaggle-live-verify ``` The public example notebooks are generated from the repo source of truth and pushed as separate Kaggle kernels: `wyattowalsh/openoppsdb-starter-notebook`, `wyattowalsh/openoppsdb-advanced-usage`, `wyattowalsh/openoppsdb-hiring-market-map`, and `wyattowalsh/openoppsdb-skills-radar`. They are read-only, internet-disabled, credential-free, and attached only to `wyattowalsh/openoppsdb`. After pushing, run `just kaggle-example-notebooks-pull-check` to pull and verify the live source bundles; `just kaggle-example-notebooks-files page_size=200` lists output files emitted by those notebook runs. Use `just kaggle-dataset-create` only for the first public upload; use `just kaggle-dataset-version` after the dataset exists. Use `just kaggle-runtime-generator-create` only for the first private manager-runtime upload; use `just kaggle-runtime-generator-version` before pushing the manager notebook after generator changes. The live write recipes stage temporary upload directories and run through the Kaggle CLI, so run `kaggle auth login` before publishing and retry with `kaggle auth login --force` if Kaggle write endpoints return `401`. They fail clearly when Kaggle credentials are unavailable and are intentionally outside CI. `just kaggle-notebook-push` and `just kaggle-example-notebooks-push` pass a one-hour Kaggle kernel timeout by default; override their `timeout` argument only for a deliberate longer maintenance run. ## CI/CD [#cicd] GitHub Actions mirrors local validation instead of inventing a second build path. CI should stay least-privilege, cache-aware, and split into independently debuggable lanes: * Python package tests and coverage through `uv`. * Lock-file consistency through `uv lock --check`. * OpenSpec strict validation for every active change. * Docs generated data, MDX/type-check, build, tests, and lint through `pnpm`. * Kaggle metadata generation for exported schema changes. * Workflow lint and whitespace checks. The broader `rtk lint` docs checklist remains an explicit maintainer recipe (`just docs-rtk-lint`) rather than a conditional CI step. When CI changes, update `Justfile`, this operations page, root README, and repo instructions in the same logical change. ## Troubleshooting [#troubleshooting] | Symptom | Check | | -------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | A source sync returns zero boards | Run `uv run openopps admin sources test --page-size 5` and check provider health. | | A board has provider hints but no jobs | Run `admin providers probe-routes` and inspect unknown route candidates. | | A new source has low yield | Run `uv run openopps admin sources yield --source ` after route probing and job sync. | | Cached data looks stale | Rerun the operation with `--refresh-cache` or purge the relevant cache namespace. | | Local DB reports missing v0.1 columns | Reset the pre-release local DB file or set `OPENOPPS_DB_URL` to a new SQLite path, then rerun `admin db init`. | | Workday sync is slow | Lower `OPENOPPS_WORKDAY_CONCURRENCY` and narrow sync with `--source`, `--board`, or `--provider workday`. | | Export output is unexpectedly empty | Confirm the database has data with `uv run openopps admin db status` and inspect filters. | | Docs navigation is stale | Update `docs/content/docs/meta.json`, then run `cd docs && pnpm types:check`. | # Providers (/docs/providers) OpenOpps v0.1 separates aggregate source adapters from job-capable provider adapters. Source adapters discover company boards; provider adapters detect or fetch postings from public board providers through the CLI and local SQLite state. Python plugins can contribute additional source adapters and job providers through the `openopps.plugins` entry point group. Plugin capabilities appear in `admin providers list` and registry surfaces when loaded successfully, and load failures, disabled entries, allow-list filters, and conflicts are visible in `plugins list`. Installed plugins are discovered by default but only executed when their entry-point names are listed in `OPENOPPS_PLUGIN_ALLOWED`, unless `OPENOPPS_PLUGIN_AUTOLOAD=true` is set. Use `examples/plugins/minimal-openopps-plugin/` as the starting template for a packaged plugin with a `pyproject.toml` entry point. ## Source Adapter Inventory [#source-adapter-inventory] ## Source Catalog Sample [#source-catalog-sample] ## Non-VC Source Families [#non-vc-source-families] OpenOpps packages a small set of low-friction non-VC source families as source adapters, not job providers. These sources add company candidates and provenance metadata; they only become job-yielding after route detection, route probing, and provider job syncs find public job-capable routes. | Source family | Packaged keys | Notes | | ----------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------------- | | Official public-company index | `sec-company-tickers` | Included listed-company backbone; SEC fair-access controls can reject generic scheduled sync environments. | | Public index CSV | `sp500`, `nasdaq100` | Included seed data for index membership; `nasdaq100` remains manual until a reviewed CSV is configured. | | Employer ranking CSV | `fortune500` | Included ranking seed data; supports embedded user-supplied CSV rows for reviewed local refreshes. | | Ecosystem landscape | `cncf-landscape` | Reads CNCF `landscape.yml` public fields and intentionally excludes logos and Crunchbase-derived fields. | Each packaged source includes taxonomy metadata such as `providerType`, `coverageMode`, `accessType`, `licenseStatus`, `refreshCadence`, and `sourceAttribution` in the source `raw_metadata` payload. ## Support Levels [#support-levels] | Level | Meaning | | ------------- | ----------------------------------------------------------------------------------------- | | `detect` | OpenOpps can preserve provider metadata and route hints but does not reliably fetch jobs. | | `jobs` | OpenOpps can fetch public postings for the provider. | | `unsupported` | The provider is known only as raw upstream metadata. | ## Job-Capable Providers [#job-capable-providers] ## Detect-Only Providers [#detect-only-providers] `manatal` and `gem` are preserved as board metadata until stable public fetching is added. Health reports group these under `notCovered` so they are visible without being treated as failed job syncs. ## Public No-Auth Board Providers [#public-no-auth-board-providers] Workable, Teamtailor, BambooHR, Rippling, and WP Job Manager are job-capable in v0.1 only through public unauthenticated board routes. Workable fetches list and per-job detail endpoints separately so `raw_listing` and `raw_detail` stay distinct for audit replay. BambooHR uses public careers endpoints such as `/careers/list` and `/careers/{job_id}/detail`; OpenOpps does not call authenticated BambooHR ATS APIs. WP Job Manager requires an explicit `/wp-json/wp/v2/job-listings` or `/jm-ajax/get_listings/` endpoint and is not inferred from arbitrary WordPress sites. ## Surplus field promotion [#surplus-field-promotion] Greenhouse list responses with `content=true` promote `metadata`, `requisition_id`, `language`, and department/office hierarchy into `provider_extras`; prospect posts without `internal_job_id` are tagged `posting_kind=prospect`. See the S1–S4 surplus taxonomy in [Data Model](/docs/data-model#ingest-surplus-taxonomy-s1s4) and `openspec/changes/ingest-data-surplus/` for the full promotion manifest. | Provider | Shipped in v0.1 | Planned (manifest) | | -------------- | --------------------------------------------------------------------------------- | ----------------------------------------- | | Greenhouse | `metadata`, `requisition_id`, `language`, department/office trees, `posting_kind` | `pay_input_ranges` (optional N+1 fetch) | | Workable | `raw_listing` / `raw_detail` split | — | | Lever | — | `categories`, epoch dates, `sections` | | Ashby | — | `isListed`, compensation, `workplaceType` | | BambooHR | — | `jobOpening`, requisition ids | | Workday | — | `postedOn`, `jobDescription` | | Rippling | — | `payRangeDetails`, `workplaceType` | | Teamtailor | — | RSS limits; optional HTML detail fetch | | WP Job Manager | — | meta keys | ## Provider Coverage [#provider-coverage] Provider coverage reports on the persisted SQLite dataset only. It does not perform live HTTP checks, source fetches, route probes, or job syncs. The deterministic smoke data proves the report shape; published real-world percentages must be measured from representative persisted source snapshots: ```bash uv run openopps providers coverage uv run openopps providers coverage --source a16z --provider any --json uv run openopps providers coverage --source a16z --provider greenhouse uv run openopps providers audit --source a16z --json uv run openopps admin sources yield --json ``` Use coverage to answer whether the local data is complete enough for analysis. The JSON report includes source, board, route, and job counts; route counts by provider, support level, and last status; executable route counts; missing route metadata counts; duplicate route skips derived from the durable route registry; job counts by provider, source, and board; non-supported provider coverage; detect-only provider examples; boards with job-capable hints but no executable route; and boards with executable routes but zero persisted jobs. `admin sources yield` reports source-family conversion from persisted records only: company candidates, canonical boards, provider hints, job-capable routes, route-ready routes, active job routes, duplicate board rate, active boards added, yield score, and taxonomy totals by provider type and access type. Coverage also reports enrichment completeness percentages from deterministic provider-field mapping. These data-quality metrics cover posting URLs, apply URLs, locations, departments, descriptions, normalized compensation or salary, remote level, and employment type. ## Route Probing [#route-probing] Some sources report provider hints without the public route token required for job fetching. Route probing tries candidate tokens derived from upstream slugs, remote ids, names, domains, and websites. Board keys such as `a16z:acme` are durable record identifiers, not provider route-token candidates.
Candidate tokens Derived from source slugs, remote ids, names, domains, and websites.
Dry-run first Route probing reports matches and unknowns without persistence by default.
Apply deliberately Add --apply only after inspecting matched route metadata.
```bash uv run openopps admin providers probe-routes --source a16z --provider any --limit 25 --json ``` By default, probing only checks routes missing token or URL metadata and does not persist results. Use `--include-existing` to recheck existing routes and `--apply` to persist matched metadata. ## Provider Health [#provider-health] Provider health samples source adapters and job-capable routes, then reports status counts: ```bash uv run openopps providers health --source a16z --provider any --limit 25 --json ``` Health checks report `active`, `empty`, `error`, `missing_route`, `not_covered`, and duplicate route skips. Add `--apply` to persist source health under `raw_metadata.health` and board-provider route health under `last_status`. Use provider health for live sampled HTTP status. Use provider coverage for offline persisted route coverage and enrichment quality. Use provider audit for candidate-provider adoption evidence. ## Board Route Registry [#board-route-registry] The `board_providers` registry is the executable route layer between discovered boards and job sync. Use it before large syncs to confirm which routes are ready: ```bash uv run openopps admin providers registry --provider any uv run openopps admin providers registry --provider any --passed-probe-only --json uv run openopps admin providers registry --provider any --include-missing --limit 50 ``` Without `--include-missing`, `admin providers registry` skips job-capable hints that still lack executable metadata, such as an Ashby board token or complete Workday CXS route. `--passed-probe-only` narrows output to routes verified by a persisted `admin providers probe-routes --apply` result. Use [Explorer](/explorer) to inspect the generated static snapshot of persisted boards, board-provider routes, latest job rows, source/provider coverage, and data-quality signals. ## Provider Limits [#provider-limits] * Workday support is limited to public postings visible through careers sites; it is not official tenant API access. * Ashby sync excludes `isListed: false` direct-link-only postings from normal output. * Overlapping board records from multiple sources are merged by company domain. Board JSON includes `source_keys` and `source_board_keys` for every source currently represented by the canonical board, and provider requests are deduped before route probes and job syncs. * Installed Python plugins are not sandboxed and run in the same process as OpenOpps. * Use `OPENOPPS_PLUGIN_ALLOWED` to opt in trusted entry-point names before plugin code executes. * Use `OPENOPPS_PLUGIN_AUTOLOAD=true` only in controlled environments where every installed `openopps.plugins` entry point is trusted.