diff --git a/README.md b/README.md index 972e623..cffbeb8 100755 --- a/README.md +++ b/README.md @@ -1,31 +1,402 @@ -# Introduction +# zxh / org-web -This repository contains code for the website https://zainezq.com, which serves as a central familial hub for me. +This repository is the source and publishing pipeline for [zainezq.com](https://zainezq.com), a personal and family website built primarily from Emacs Org-mode files. It contains the written content, the Emacs Lisp build system, browser assets, generated indexes, playful interactive pages, and the Gitea Actions workflow that publishes the site. -The site is written using Emacs and Org-mode, and exported using the `org-publish` functionality. See the `build-site.el` script for details on how the site is built and published. +This README is the central technical guide for the repository: start here for architecture, authoring, builds, frontend work, testing, and deployment. -Asset structure and frontend boundaries are documented in `docs/assets-architecture.md`. +## System overview -## Local authoring UI +The application has three boundaries: -The authoring UI now lives in `/home/zaine/master-folder/org-platform/authoring-service`. +1. **Site source (this repository):** Org/Markdown content, Emacs Lisp publishing code, CSS, JavaScript, images, and build tooling. +2. **Static output (`output/`):** generated HTML, copied assets, and `search-index.json`. This is the deployable website, not the source of truth. +3. **Runtime services (outside this repository):** the authoring service and the same-origin APIs used by interactive pages. The static site remains readable without them, but the relevant dynamic features will not work. -Run `make author` from either repository to install and start the user systemd service, then open `http://127.0.0.1:8765` to create or edit blog and post Org files from a browser. When deployed behind the reverse proxy, use `https://author.zainezq.com`. +```mermaid +flowchart LR + Author["Author or authoring service"] --> Source["Org and Markdown source"] + Source --> Emacs["Emacs org-publish pipeline"] + Assets["CSS, JavaScript, images, fonts"] --> Emacs + Emacs --> HTML["output/: HTML and assets"] + HTML --> Search["Python search indexer"] + Search --> Deploy["Static site at zainezq.com"] + Deploy <-->|"same-origin /api requests"| APIs["Runtime APIs outside this repo"] +``` -Useful service commands: +## Repository map -- `make author-restart` restarts the service and waits for the page API to respond -- `make author-stop` stops the service -- `make author-status` shows systemd status and the editable page count -- `make author-logs` shows recent journal logs +| Path | Purpose | Source or generated? | +| --- | --- | --- | +| `index.org` | Home page source | Source | +| `blogs/` | Dated personal writing, grouped by year/month | Source, except generated list pages | +| `posts/` | Longer-lived posts; `posts/career/` has its own index | Source, except generated list pages | +| `home/` | Utility and dashboard-style pages | Source, except `categories.org` | +| `play/` | Interactive and experimental pages | Source | +| `lima/` | Org or Markdown pages with special attachment handling | Source, except `lima-list.org` | +| `assets/` | Browser code, styles, fonts, images, and content manifests | Mostly source; some manifests/scripts are generated externally | +| `lisp/` | Modular Emacs Lisp publishing system | Source | +| `lisp/tests/` | ERT tests for the publishing logic | Source | +| `tags/` | Per-tag Org pages created during a build | Generated | +| `recently-updated.org` | Recent-content list created during a build | Generated | +| `sitemap.org` and `*-list.org` | Sitemap/list sources created by `org-publish` | Generated | +| `output/` | Complete publish output | Generated | +| `.org-timestamps/` | Incremental publish cache | Generated | +| `.packages/` | Repository-local Emacs package directory | Generated/cache | +| `build-logs/` | Gitea build monitoring script, cache, and logs | Mixed; script is source, logs/cache are runtime data | +| `docs/` | Historical/design notes for individual features | Supplemental; this README is the central engineering guide | -The editor writes normal `.org` pages with the metadata used by the publishing pipeline: +## Build architecture -- blogs default to `blogs/YYYY/MM-month/slug.org` -- posts default to `posts/slug.org`, or `posts/section/slug.org` when a section is provided -- new pages can use an explicit path under `blogs/` or `posts/` -- saving a page starts `emacs -Q --script build-site.el` and then regenerates the search index +`build-site.el` is the entry point. It establishes absolute source/output roots, bootstraps `htmlize`, configures Org, loads the modules in `lisp/`, defines the publishing projects, and runs the build. -Lima pages are Markdown files under `lima/`. Selecting the Lima page type shows a Markdown toolbar for common formatting and an attachment button. Uploaded images and videos are stored under `assets/images/hzone/YYYY/MM/` and inserted into the Markdown page. +```mermaid +flowchart TD + Entry["build-site.el"] --> Core["build-core.el: packages, Org, logging"] + Core --> Modules["Load build modules"] + Modules --> Projects["build-projects.el: org-publish projects"] + Projects --> Prepare["Validate output and timestamp cache"] + Prepare --> Inputs["Generate tag pages and comment manifest"] + Inputs --> Publish["org-publish-all"] + Publish --> OrgHTML["Org/Markdown to HTML"] + Publish --> AssetCopy["Copy assets"] + OrgHTML --> Sync["Final asset sync"] + AssetCopy --> Sync + Sync --> Summary["Build summary"] +``` -Saves enqueue builds. The editor remains usable while publishing runs, and the queue section shows the current build, pending builds, and recent build results. +### Build modules + +| Module | Responsibility | +| --- | --- | +| `build-core.el` | Logging, timing, local package bootstrap, and base Org configuration | +| `build-assets.el` | Shared CSS/JS `` manifest, cache-busted asset URLs, and asset synchronization | +| `build-html.el` | Custom HTML backend, page shell, Mermaid export, macros, body classes, tags, and comments | +| `build-projects.el` | `org-publish-project-alist` for the main site, posts, blogs, career, tags, Lima, categories, and assets | +| `build-runner.el` | Cache integrity, generated inputs, publishing, and build metrics | +| `build-lima.el` | Lima Org/Markdown conversion and neighboring attachment copying | +| `build-comments.el` | Comment opt-in detection, comment HTML injection, and comment-page manifest generation | +| `sitemaps.el` | Main Mermaid sitemap and specialized blog/post/career/category lists | +| `tags.el` | Tag discovery, generated tag pages, and tag markup in exported HTML | +| `recently-updated.el` | Recent page list generation using `#+DATE` with mtime fallback | +| `sidenotes.el` | Sidenote suppression rules for special page layouts | + +The main `org-main` project recursively publishes every Org file, while narrower projects provide specialized sitemap behavior and options. That overlap is intentional: Org's timestamp cache prevents unnecessary repeated work in normal incremental builds. + +### Generated inputs and outputs + +Before publishing, the build writes: + +- `tags/*.org` from `#+FILETAGS`/`#+TAGS` in posts and blogs. +- `assets/content/comment-pages.json` from pages with `#+COMMENTS: t`. +- `recently-updated.org` from the 26 most recent eligible Org pages. +- sitemap/list Org files through `org-publish`. + +After publishing, `search-index-json.py` walks generated HTML, strips script/style markup with Beautiful Soup, and writes `output/search-index.json`. The browser builds a Lunr index from that JSON at page load. + +```mermaid +flowchart LR + Metadata["Org metadata"] --> Tags["tags/*.org"] + Metadata --> Comments["comment-pages.json"] + Metadata --> Recent["recently-updated.org"] + Tags --> Publish["HTML publish"] + Comments --> Publish + Recent --> Publish + Publish --> Indexer["search-index-json.py"] + Indexer --> SearchJSON["output/search-index.json"] + SearchJSON --> Lunr["Browser Lunr index"] +``` + +## Prerequisites + +For a full local build, install: + +- GNU Emacs with Org mode (Org is bundled with normal Emacs distributions). +- Python 3 with `venv` support. +- `pandoc`, required when publishing Markdown files under `lima/`. +- Network access on the first build if `htmlize` is not already cached in `.packages/`. + +Python packages are listed in `requirements.txt` and installed into `.venv` by Make. The default `make` workflow also invokes a machine-specific permission script with `sudo`; use the direct build commands below on machines that do not have that script. + +## Build and development commands + +### Portable local build + +```sh +emacs -Q --script build-site.el +python3 -m venv .venv +.venv/bin/pip install -r requirements.txt +.venv/bin/python search-index-json.py +``` + +The generated site is in `output/`. Serve it over HTTP so fetch-based features work correctly, for example: + +```sh +python3 -m http.server 8000 --directory output +``` + +Then open `http://127.0.0.1:8000`. Dynamic `/api/...` features require the external runtime services or an equivalent local proxy. + +### Make targets + +| Command | Effect | +| --- | --- | +| `make build` | Run the Emacs publishing pipeline | +| `make search` | Create/update `output/search-index.json`, creating `.venv` when needed | +| `make` | Set machine-specific permissions, remove `output/`, build, index search, and remove `.venv` | +| `make clean-output` | Delete the configured output directory | +| `make test-elisp` | Run the Emacs Lisp ERT suite | +| `make test` | Run tests in the sibling authoring-service repository | +| `make norm` | Move Emacs backup files into `backups/` | +| `make help` | List supported targets | + +### Build environment variables + +| Variable | Meaning | +| --- | --- | +| `SITE_OUTPUT_DIR=/path` | Publish somewhere other than `output/`; the Python indexer respects it too | +| `SITE_FORCE=1` | Republish every file instead of using Org timestamps | +| `SITE_DRY=1` | Generate/prepare inputs but skip `org-publish-all` | +| `SITE_CLEAN=1` | Delete output and `.org-timestamps/`, then force a full publish | + +The build checks that assets and an entry page exist in the output. If the output is missing or incomplete while timestamps remain, it clears the timestamp cache and forces a full publish to avoid a falsely successful incremental build. + +## Authoring content + +### Org pages + +A typical page is: + +```org +#+TITLE: Page title +#+DATE: <2026-07-12 Sun 12:00> +#+FILETAGS: :topic:notes: +#+OPTIONS: num:nil toc:t +#+COMMENTS: t +#+SLUG: stable-page-slug + +* First heading + +Content goes here. +``` + +The important metadata is: + +- `#+TITLE`: displayed title and sitemap/search label. +- `#+DATE`: canonical publication time and ordering source. Recently updated falls back to filesystem modification time when absent or invalid. +- `#+FILETAGS`: colon-delimited tags used for category/tag pages and rendered under the page title. +- `#+OPTIONS`: normal Org export controls such as `toc:nil`, `num:nil`, or `title:nil`. +- `#+COMMENTS: t`: injects the comment UI and adds the page to the comment manifest. +- `#+SLUG`: stable comment API identifier; defaults to the filename when omitted. +- `#+NO_SIDENOTES`: any non-empty value adds the `no-sidenotes` body class. The `NO_SIDENOTES` or `KANBAN` file tag and a heading containing `KANBAN` do the same. + +Use root-relative URLs (`/assets/...`) for shared site assets. Relative Org file links are rewritten to their corresponding HTML links by the exporter. + +### Supported export helpers + +Mermaid diagrams can be written directly in Org: + +```org +#+begin_src mermaid +flowchart LR + A[Write] --> B[Publish] +#+end_src +``` + +The custom HTML backend emits a Mermaid container, and the shared browser initializer renders it. The build also provides these Org macros: + +- `{{{sidenote(id,text)}}}` +- `{{{epigraph(quote,attribution)}}}` and related epigraph variants +- `{{{kbd(text)}}}` +- `{{{margimg(url,alt,caption)}}}` +- `{{{countdown(datetime,label)}}}` + +### Where new pages belong + +- Blog entry: `blogs/YYYY/MM-month/slug.org`. +- General post: `posts/slug.org`. +- Career post: `posts/career/slug.org`. +- Utility/dashboard page: `home/slug.org`. +- Interactive experiment: `play/slug.org`, usually paired with `assets/scripts/pages/slug.js` and `assets/styles/pages/slug.css`. +- Lima page: `.org` or `.md` under `lima/`. + +Do not manually maintain generated list, category, recent, tag, or output files; rebuild them from source. + +### Lima Markdown publishing + +Markdown under `lima/` is converted to temporary Org with Pandoc, given generated title/date/comment metadata, and then exported through the normal HTML backend. The first Markdown H1 becomes the page title. Sibling directories matching `.attachments.*` are copied beside the published page. Org files in `lima/` use the same attachment-copy behavior without Pandoc. + +## Frontend architecture + +All static browser assets live under `assets/` and are copied to `output/assets/`. `lisp/build-assets.el` is the authoritative list of globally loaded styles and scripts; it appends deterministic mtime/size query strings for cache busting. + +### Styles + +The modular CSS hierarchy is: + +```text +assets/styles/ +├── themes/ design tokens and themes +├── base/ element defaults +├── layout/ site shell and layout +├── components/ reusable UI components +├── pages/ page-specific presentation +├── features/ optional feature presentation +├── utilities/ responsive/utility rules +└── vendor/ third-party CSS +``` + +Top-level CSS files are compatibility manifests that import the modular files. Prefer editing the modular source, while preserving the top-level entry points until old handwritten references have been removed. + +### Scripts + +The intended JavaScript boundaries are: + +```text +assets/scripts/ +├── application/ cross-page use-case orchestration +├── ui/ DOM presentation behavior +├── pages/ page-specific applications +├── features/ optional site-wide features +└── vendor/ vendored third-party libraries +``` + +The shared script list is loaded on every page, so each module must safely exit when its expected DOM is absent. Page modules should own page composition; reusable browser behavior belongs in `ui/` or `features/`. Top-level duplicate filenames are legacy loaders or compatibility copies—edit the modular file referenced by `build-assets.el` unless a page explicitly loads the legacy path. + +Third-party browser dependencies are vendored: Lunr, Mermaid, BiggerPicture, and SVG Pan Zoom. Avoid editing minified vendor files directly. + +## Dynamic features and service boundaries + +The published HTML is static, but some browser modules rely on same-origin APIs: + +| Feature | Browser module | Runtime endpoint | +| --- | --- | --- | +| Comments and replies | `features/comments.js` | `GET /api/comments/:slug`, `POST /api/comments` | +| Notes board | `pages/notes.js` | `/api/notes` | +| Wird tracker | `pages/wird-tracker.js` | `/api/wird`, `/api/wird/motalah`, `/api/calibre/books` | +| Competency board | `pages/competency-status-board.js` | `/api/competencies/items...` | +| RPG saves | `pages/play.js`, `pages/ash-below-lake.js` | `/api/play/rpg/save/:slot`, with local-storage fallback | + +These APIs are not implemented here. Production routing must serve `output/` and proxy `/api/` to the appropriate backend. When changing an endpoint contract, coordinate the static client and its external service. + +The hidden-details feature uses browser `localStorage`/`sessionStorage` for visit and discovery state and may call Open-Meteo after explicit geolocation interaction. Its friendly content source is `assets/content/hidden-details.json`; `assets/scripts/features/hidden-details.js` is generated by the external hidden-narrative authoring workflow. Treat both files as a pair and do not hand-edit only the generated JavaScript. + +## Authoring service + +The browser authoring UI is a separate sibling repository at: + +```text +/home/zaine/master-folder/org-platform/authoring-service +``` + +It creates and edits the normal source files described above, stores Lima uploads under `assets/images/hzone/YYYY/MM/`, and enqueues site builds after saves. It is available locally at `http://127.0.0.1:8765` and, behind the production reverse proxy, at `https://author.zainezq.com`. + +The Makefile delegates these commands to that repository: + +- `make author` / `make author-start` +- `make author-install` +- `make author-stop` +- `make author-restart` +- `make author-health` +- `make author-status` +- `make author-diagnostics` +- `make author-logs` + +They depend on the sibling repository existing at the hard-coded path and use its user systemd service. + +## Testing and verification + +Run the publishing tests: + +```sh +make test-elisp +``` + +For a documentation/content-only change, a dry run validates package loading and generated inputs without rewriting all HTML: + +```sh +SITE_DRY=1 emacs -Q --script build-site.el +``` + +For build or frontend changes, use a clean build, generate the search index, serve `output/`, and inspect the relevant pages in a browser: + +```sh +SITE_CLEAN=1 emacs -Q --script build-site.el +.venv/bin/python search-index-json.py +python3 -m http.server 8000 --directory output +``` + +Verify at least: + +- the home page and global navigation; +- a blog/post with a table of contents, tags, sidenotes, and comments as applicable; +- the relevant special page under `home/` or `play/`; +- search results and keyboard navigation; +- browser console/network failures; +- the generated sitemap and tag/category pages. + +## Deployment and CI + +`.gitea/workflows/build.yml` runs on pushes to `main`, nightly at midnight, and manual dispatch. Push events that only change the hidden-details JSON/generated JS pair are ignored to prevent a build loop. + +```mermaid +sequenceDiagram + participant Git as Gitea main + participant Runner as org-web runner + participant Build as Site build + participant Monitor as PowerShell monitor + Git->>Runner: push, schedule, or manual dispatch + opt scheduled or manual + Runner->>Git: commit and push hidden-detail changes if present + end + Runner->>Build: SITE_CLEAN=1 make + Build-->>Runner: static output and logs + Runner->>Monitor: inspect result and authoring tests + Monitor-->>Runner: Discord notification +``` + +The runner is host-specific. It expects: + +- the checkout at `/home/zaine/master-folder/org-platform/org_web`; +- output at that repository's `output/` directory; +- the permission helper referenced by the Makefile; +- PowerShell at `/snap/bin/pwsh`; +- the sibling authoring-service repository; +- Gitea/Discord credentials supplied as Actions secrets. + +The workflow's monitor writes diagnostics and uses `DISCORD_WEBHOOK_URL` and `API_KEY`. Never commit those values or other service credentials. + +## Safe change checklist + +1. Edit source content, modular assets, or `lisp/`; do not edit `output/` as the source of a fix. +2. Check whether the file is generated or a compatibility wrapper before changing it. +3. Add new globally loaded assets to `lisp/build-assets.el`; keep page-only behavior scoped to its page when possible. +4. Preserve stable `#+SLUG` values on comment-enabled pages. +5. Run `make test-elisp` and the smallest representative build/browser checks for the change. +6. Review `git status` carefully: builds intentionally update generated Org files and manifests, while local logs and caches may also change. +7. Commit only the intended source and generated artifacts required by the workflow. + +## Troubleshooting + +### A build skips files that are missing from `output/` + +Run with `SITE_CLEAN=1`. The normal integrity check catches a missing asset directory or entry page, but it does not validate every generated page. + +### `htmlize` cannot be installed + +The first build needs access to MELPA/ELPA unless `.packages/` already contains it. Restore network access or provide the cached package; the build cannot continue without syntax-highlighting support. + +### Lima Markdown fails to publish + +Confirm `pandoc` is installed and available on `PATH`. Plain Org publishing does not require Pandoc. + +### Search is unavailable + +Confirm `output/search-index.json` exists and the site is served over HTTP. Run `make search` after publishing HTML. + +### Interactive data does not load locally + +Static file serving does not provide `/api/...`. Run/proxy the relevant external services, or expect those features to show their empty/error state. + +### The full `make` command fails outside the production host + +`make` runs a hard-coded permission helper with `sudo`. Use `make build` followed by `make search`, or the portable commands above, on another machine.