org_web/README.md

# zxh / org-web

This repository is the source and publishing pipeline for [zainezq.com](https://zainezq.com), a personal and family website built primarily from Emacs Org-mode files. It contains the written content, the Emacs Lisp build system, browser assets, generated indexes, playful interactive pages, and the Gitea Actions workflow that publishes the site.

This README is the central technical guide for the repository: start here for architecture, authoring, builds, frontend work, testing, and deployment.

## System overview

The application has three boundaries:

1. **Site source (this repository):** Org/Markdown content, Emacs Lisp publishing code, CSS, JavaScript, images, and build tooling.
2. **Static output (**`output/`**):** generated HTML, copied assets, and `search-index.json`. This is the deployable website, not the source of truth.
3. **Runtime services (outside this repository):** the authoring service and the same-origin APIs used by interactive pages. The static site remains readable without them, but the relevant dynamic features will not work.

```mermaid
flowchart LR
  Author["Author or authoring service"] --> Source["Org and Markdown source"]
  Source --> Emacs["Emacs org-publish pipeline"]
  Assets["CSS, JavaScript, images, fonts"] --> Emacs
  Emacs --> HTML["output/: HTML and assets"]
  HTML --> Search["Python search indexer"]
  Search --> Deploy["Static site at zainezq.com"]
  Deploy <-->|"same-origin /api requests"| APIs["Runtime APIs outside this repo"]
```

## Repository map

| Path                       | Purpose                                                    | Source or generated?                                           |
|----------------------------|------------------------------------------------------------|----------------------------------------------------------------|
| `index.org`                  | Home page source                                           | Source                                                         |
| `blogs/`                     | Dated personal writing, grouped by year/month              | Source, except generated list pages                            |
| `posts/`                     | Longer-lived posts; `posts/career/` has its own index        | Source, except generated list pages                            |
| `home/`                      | Utility and dashboard-style pages                          | Source, except `categories.org`                                  |
| `play/`                      | Interactive and experimental pages                         | Source                                                         |
| `rl/`                        | Server-synced Resource Loader SPA                          | Source                                                         |
| `lima/`                      | Org or Markdown pages with special attachment handling     | Source, except `lima-list.org`                                   |
| `assets/`                    | Browser code, styles, fonts, images, and content manifests | Mostly source; some manifests/scripts are generated externally |
| `lisp/`                      | Modular Emacs Lisp publishing system                       | Source                                                         |
| `lisp/tests/`                | ERT tests for the publishing logic                         | Source                                                         |
| `tags/`                      | Per-tag Org pages created during a build                   | Generated                                                      |
| `recently-updated.org`       | Recent-content list created during a build                 | Generated                                                      |
| `sitemap.org` and `*-list.org` | Sitemap/list sources created by `org-publish`                | Generated                                                      |
| `output/`                    | Complete publish output                                    | Generated                                                      |
| `.org-timestamps/`           | Incremental publish cache                                  | Generated                                                      |
| `.packages/`                 | Repository-local Emacs package directory                   | Generated/cache                                                |
| `build-logs/`                | Gitea build monitoring script, cache, and logs             | Mixed; script is source, logs/cache are runtime data           |
| `docs/`                      | Historical/design notes for individual features            | Supplemental; this README is the central engineering guide     |

## Build architecture

`build-site.el` is the entry point. It establishes absolute source/output roots, bootstraps `htmlize`, configures Org, loads the modules in `lisp/`, defines the publishing projects, and runs the build.

```mermaid
flowchart TD
  Entry["build-site.el"] --> Core["build-core.el: packages, Org, logging"]
  Core --> Modules["Load build modules"]
  Modules --> Projects["build-projects.el: org-publish projects"]
  Projects --> Prepare["Validate output and timestamp cache"]
  Prepare --> Inputs["Generate tag pages and comment manifest"]
  Inputs --> Publish["org-publish-all"]
  Publish --> OrgHTML["Org/Markdown to HTML"]
  Publish --> AssetCopy["Copy assets"]
  OrgHTML --> Sync["Final asset sync"]
  AssetCopy --> Sync
  Sync --> Summary["Build summary"]
```

### Build modules

| Module              | Responsibility                                                                                        |
|---------------------|-------------------------------------------------------------------------------------------------------|
| `build-core.el`       | Logging, timing, local package bootstrap, and base Org configuration                                  |
| `build-assets.el`     | Shared CSS/JS `<head>` manifest, cache-busted asset URLs, and asset synchronization                     |
| `build-html.el`       | Custom HTML backend, page shell, Mermaid export, macros, body classes, tags, and comments             |
| `build-projects.el`   | `org-publish-project-alist` for the main site, posts, blogs, career, tags, Lima, categories, and assets |
| `build-runner.el`     | Cache integrity, generated inputs, publishing, and build metrics                                      |
| `build-lima.el`       | Lima Org/Markdown conversion and neighboring attachment copying                                       |
| `build-comments.el`   | Comment opt-in detection, comment HTML injection, and comment-page manifest generation                |
| `sitemaps.el`         | Main Mermaid sitemap and specialized blog/post/career/category lists                                  |
| `tags.el`             | Tag discovery, generated tag pages, and tag markup in exported HTML                                   |
| `recently-updated.el` | Recent page list generation using `#+DATE` with mtime fallback                                          |
| `sidenotes.el`        | Sidenote suppression rules for special page layouts                                                   |

The main `org-main` project recursively publishes every Org file, while narrower projects provide specialized sitemap behavior and options. That overlap is intentional: Org's timestamp cache prevents unnecessary repeated work in normal incremental builds.

### Generated inputs and outputs

Before publishing, the build writes:

- `tags/*.org` from `#+FILETAGS`/`#+TAGS` in posts and blogs.
- `assets/content/comment-pages.json` from pages with `#+COMMENTS: t`.
- `recently-updated.org` from the 26 most recent eligible Org pages.
- sitemap/list Org files through `org-publish`.

After publishing, `search-index-json.py` walks generated HTML, strips script/style markup with Beautiful Soup, and writes `output/search-index.json`. The browser builds a Lunr index from that JSON at page load.

```mermaid
flowchart LR
  Metadata["Org metadata"] --> Tags["tags/*.org"]
  Metadata --> Comments["comment-pages.json"]
  Metadata --> Recent["recently-updated.org"]
  Tags --> Publish["HTML publish"]
  Comments --> Publish
  Recent --> Publish
  Publish --> Indexer["search-index-json.py"]
  Indexer --> SearchJSON["output/search-index.json"]
  SearchJSON --> Lunr["Browser Lunr index"]
```

## Prerequisites

For a full local build, install:

- GNU Emacs with Org mode (Org is bundled with normal Emacs distributions).
- Python 3 with `venv` support.
- `pandoc`, required when publishing Markdown files under `lima/`.
- Network access on the first build if `htmlize` is not already cached in `.packages/`.

Python packages are listed in `requirements.txt` and installed into `.venv` by Make. The default `make` workflow also invokes a machine-specific permission script with `sudo`; use the direct build commands below on machines that do not have that script.

## Build and development commands

### Portable local build

```sh
emacs -Q --script build-site.el
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python search-index-json.py
```

The generated site is in `output/`. Serve it over HTTP so fetch-based features work correctly, for example:

```sh
python3 -m http.server 8000 --directory output
```

Then open `http://127.0.0.1:8000`. Dynamic `/api/...` features require the external runtime services or an equivalent local proxy.

### Make targets

| Command           | Effect                                                                                  |
|-------------------|-----------------------------------------------------------------------------------------|
| `make build`        | Run the Emacs publishing pipeline                                                       |
| `make search`       | Create/update `output/search-index.json`, creating `.venv` when needed                      |
| `make`              | Set machine-specific permissions, remove `output/`, build, index search, and remove `.venv` |
| `make clean-output` | Delete the configured output directory                                                  |
| `make test-elisp`   | Run the Emacs Lisp ERT suite                                                            |
| `make test`         | Run tests in the sibling authoring-service repository                                   |
| `make norm`         | Move Emacs backup files into `backups/`                                                   |
| `make help`         | List supported targets                                                                  |

### Build environment variables

| Variable              | Meaning                                                                  |
|-----------------------|--------------------------------------------------------------------------|
| `SITE_OUTPUT_DIR=/path` | Publish somewhere other than `output/`; the Python indexer respects it too |
| `SITE_FORCE=1`          | Republish every file instead of using Org timestamps                     |
| `SITE_DRY=1`            | Generate/prepare inputs but skip `org-publish-all`                         |
| `SITE_CLEAN=1`          | Delete output and `.org-timestamps/`, then force a full publish            |

The build checks that assets and an entry page exist in the output. If the output is missing or incomplete while timestamps remain, it clears the timestamp cache and forces a full publish to avoid a falsely successful incremental build.

## Authoring content

### Org pages

A typical page is:

```org
#+TITLE: Page title
#+DATE: <2026-07-12 Sun 12:00>
#+FILETAGS: :topic:notes:
#+OPTIONS: num:nil toc:t
#+COMMENTS: t
#+SLUG: stable-page-slug

* First heading

Content goes here.
```

The important metadata is:

- `#+TITLE`: displayed title and sitemap/search label.
- `#+DATE`: canonical publication time and ordering source. Recently updated falls back to filesystem modification time when absent or invalid.
- `#+FILETAGS`: colon-delimited tags used for category/tag pages and rendered under the page title.
- `#+OPTIONS`: normal Org export controls such as `toc:nil`, `num:nil`, or `title:nil`.
- `#+COMMENTS: t`: injects the comment UI and adds the page to the comment manifest.
- `#+SLUG`: stable comment API identifier; defaults to the filename when omitted.
- `#+NO_SIDENOTES`: any non-empty value adds the `no-sidenotes` body class. The `NO_SIDENOTES` or `KANBAN` file tag and a heading containing `KANBAN` do the same.

Use root-relative URLs (`/assets/...`) for shared site assets. Relative Org file links are rewritten to their corresponding HTML links by the exporter.

### Supported export helpers

Mermaid diagrams can be written directly in Org:

```org
#+begin_src mermaid
flowchart LR
  A[Write] --> B[Publish]
#+end_src
```

The custom HTML backend emits a Mermaid container, and the shared browser initializer renders it. The build also provides these Org macros:

- `{{{sidenote(id,text)}}}`
- `{{{epigraph(quote,attribution)}}}` and related epigraph variants
- `{{{kbd(text)}}}`
- `{{{margimg(url,alt,caption)}}}`
- `{{{countdown(datetime,label)}}}`

### Where new pages belong

- Blog entry: `blogs/YYYY/MM-month/slug.org`.
- General post: `posts/slug.org`.
- Career post: `posts/career/slug.org`.
- Utility/dashboard page: `home/slug.org`.
- Interactive experiment: `play/slug.org`, usually paired with `assets/scripts/pages/slug.js` and `assets/styles/pages/slug.css`.
- Lima page: `.org` or `.md` under `lima/`.

Do not manually maintain generated list, category, recent, tag, or output files; rebuild them from source.

### Lima Markdown publishing

Markdown under `lima/` is converted to temporary Org with Pandoc, given generated title/date/comment metadata, and then exported through the normal HTML backend. The first Markdown H1 becomes the page title. Sibling directories matching `.attachments.*` are copied beside the published page. Org files in `lima/` use the same attachment-copy behavior without Pandoc.

## Frontend architecture

All static browser assets live under `assets/` and are copied to `output/assets/`. `lisp/build-assets.el` is the authoritative list of globally loaded styles and scripts; it appends deterministic mtime/size query strings for cache busting.

### Styles

The modular CSS hierarchy is:

```text
assets/styles/
├── themes/       design tokens and themes
├── base/         element defaults
├── layout/       site shell and layout
├── components/   reusable UI components
├── pages/        page-specific presentation
├── features/     optional feature presentation
├── utilities/    responsive/utility rules
└── vendor/       third-party CSS
```

Top-level CSS files are compatibility manifests that import the modular files. Prefer editing the modular source, while preserving the top-level entry points until old handwritten references have been removed.

### Scripts

The intended JavaScript boundaries are:

```text
assets/scripts/
├── application/  cross-page use-case orchestration
├── ui/           DOM presentation behavior
├── pages/        page-specific applications
├── features/     optional site-wide features
└── vendor/       vendored third-party libraries
```

The shared script list is loaded on every page, so each module must safely exit when its expected DOM is absent. Page modules should own page composition; reusable browser behavior belongs in `ui/` or `features/`. Top-level duplicate filenames are legacy loaders or compatibility copies—edit the modular file referenced by `build-assets.el` unless a page explicitly loads the legacy path.

Third-party browser dependencies are vendored: Lunr, Mermaid, BiggerPicture, and SVG Pan Zoom. Avoid editing minified vendor files directly.

## Dynamic features and service boundaries

The published HTML is static, but some browser modules rely on same-origin APIs:

| Feature              | Browser module                   | Runtime endpoint                                 |
|----------------------|----------------------------------|--------------------------------------------------|
| Comments and replies | `features/comments.js`             | `GET /api/comments/:slug`, `POST /api/comments`      |
| Notes board          | `pages/notes.js`                   | `/api/notes`                                       |
| Wird tracker         | `pages/wird-tracker.js`            | `/api/wird`, `/api/wird/motalah`, `/api/calibre/books` |
| Competency board     | `pages/competency-status-board.js` | `/api/competencies/items...`                       |
| Resource Loader      | `pages/resource-loader.js`          | `/api/resource-loader`, `/api/resource-loader/thumbnails/:id` |

These APIs are not implemented here. Production routing must serve `output/` and proxy `/api/` to the appropriate backend. When changing an endpoint contract, coordinate the static client and its external service.

The hidden-details feature uses browser `localStorage`/`sessionStorage` for visit and discovery state and may call Open-Meteo after explicit geolocation interaction. Its friendly content source is `assets/content/hidden-details.json`; `assets/scripts/features/hidden-details.js` is generated by the external hidden-narrative authoring workflow. Treat both files as a pair and do not hand-edit only the generated JavaScript.

The Resource Loader at `/rl/` uses the same-origin `/api/resource-loader` backend as its sole metadata source, with ETags protecting concurrent edits. There is no browser metadata fallback. Generated cover thumbnails use `/api/resource-loader/thumbnails/:id` and are removed from local storage after upload. IndexedDB under `zxh-resource-loader` retains only imported PDF binaries on the device where they were added. JSON exports contain metadata and reading sessions, not PDF binaries. Its page code lives in `assets/scripts/pages/resource-loader.js`; the separately testable schema/progress/sync operations live in `resource-loader-model.js` and run with `make test-js`.

## Authoring service

The browser authoring UI is a separate sibling repository at:

```text
/home/zaine/master-folder/org-platform/authoring-service
```

It creates and edits the normal source files described above, stores Lima uploads under `assets/images/hzone/YYYY/MM/`, and enqueues site builds after saves. It is available locally at `http://127.0.0.1:8765` and, behind the production reverse proxy, at `https://author.zainezq.com`.

The Makefile delegates these commands to that repository:

- `make author` / `make author-start`
- `make author-install`
- `make author-stop`
- `make author-restart`
- `make author-health`
- `make author-status`
- `make author-diagnostics`
- `make author-logs`

They depend on the sibling repository existing at the hard-coded path and use its user systemd service.

## Testing and verification

Run the publishing tests:

```sh
make test-elisp
```

For a documentation/content-only change, a dry run validates package loading and generated inputs without rewriting all HTML:

```sh
SITE_DRY=1 emacs -Q --script build-site.el
```

For build or frontend changes, use a clean build, generate the search index, serve `output/`, and inspect the relevant pages in a browser:

```sh
SITE_CLEAN=1 emacs -Q --script build-site.el
.venv/bin/python search-index-json.py
python3 -m http.server 8000 --directory output
```

Verify at least:

- the home page and global navigation;
- a blog/post with a table of contents, tags, sidenotes, and comments as applicable;
- the relevant special page under `home/` or `play/`;
- search results and keyboard navigation;
- browser console/network failures;
- the generated sitemap and tag/category pages.

## Deployment and CI

`.gitea/workflows/build.yml` runs on pushes to `main`, nightly at midnight, and manual dispatch. Push events that only change the hidden-details JSON/generated JS pair are ignored to prevent a build loop.

```mermaid
sequenceDiagram
  participant Git as Gitea main
  participant Runner as org-web runner
  participant Build as Site build
  participant Monitor as PowerShell monitor
  Git->>Runner: push, schedule, or manual dispatch
  opt scheduled or manual
    Runner->>Git: commit and push hidden-detail changes if present
  end
  Runner->>Build: SITE_CLEAN=1 make
  Build-->>Runner: static output and logs
  Runner->>Monitor: inspect result and authoring tests
  Monitor-->>Runner: Discord notification
```

The runner is host-specific. It expects:

- the checkout at `/home/zaine/master-folder/org-platform/org_web`;
- output at that repository's `output/` directory;
- the permission helper referenced by the Makefile;
- PowerShell at `/snap/bin/pwsh`;
- the sibling authoring-service repository;
- Gitea/Discord credentials supplied as Actions secrets.

The workflow's monitor writes diagnostics and uses `DISCORD_WEBHOOK_URL` and `API_KEY`. Never commit those values or other service credentials.

## Safe change checklist

1. Edit source content, modular assets, or `lisp/`; do not edit `output/` as the source of a fix.
2. Check whether the file is generated or a compatibility wrapper before changing it.
3. Add new globally loaded assets to `lisp/build-assets.el`; keep page-only behavior scoped to its page when possible.
4. Preserve stable `#+SLUG` values on comment-enabled pages.
5. Run `make test-elisp` and the smallest representative build/browser checks for the change.
6. Review `git status` carefully: builds intentionally update generated Org files and manifests, while local logs and caches may also change.
7. Commit only the intended source and generated artifacts required by the workflow.

## Troubleshooting

### A build skips files that are missing from `output/`

Run with `SITE_CLEAN=1`. The normal integrity check catches a missing asset directory or entry page, but it does not validate every generated page.

### `htmlize` cannot be installed

The first build needs access to MELPA/ELPA unless `.packages/` already contains it. Restore network access or provide the cached package; the build cannot continue without syntax-highlighting support.

### Lima Markdown fails to publish

Confirm `pandoc` is installed and available on `PATH`. Plain Org publishing does not require Pandoc.

### Search is unavailable

Confirm `output/search-index.json` exists and the site is served over HTTP. Run `make search` after publishing HTML.

### Interactive data does not load locally

Static file serving does not provide `/api/...`. Run/proxy the relevant external services, or expect those features to show their empty/error state.

### The full `make` command fails outside the production host

`make` runs a hard-coded permission helper with `sudo`. Use `make build` followed by `make search`, or the portable commands above, on another machine.