gitea runners

Career Concepts/20260101010049-concepts.org

@@ -10,7 +10,9 @@ In this node lies notes relating to theoretical concepts tying with maths and/or
 * Microlise:
 ** ESS
 - [[id:abe43fdc-e90d-4c2c-9320-cc7929f0c99a][ESS]]
+- [[id:0b6e7dc0-1171-4dc4-982e-a5d95765e09c][SEB Search Improvement — Couchbase Caching]]
 
+  
 * Mathematical Concepts
 
 Part of being a software engineer is having a good grasp of mathematical concepts. Here are some notes on various mathematical concepts that I find useful.

Career Concepts/20260505121135-seb_search_improvements.org

@@ -0,0 +1,177 @@
+:PROPERTIES:
+:ID:       0b6e7dc0-1171-4dc4-982e-a5d95765e09c
+:END:
+#+TITLE: SEB Search Improvement — Couchbase Caching
+#+filetags: :couchbase:kafka:
+#+DATE: 2025-06-18
+
+* Overview
+
+DBAs are seeing high CPU and memory usage on SQL production servers due to SEB
+Search. Free text search uses the =LIKE= operator against RouteID, JourneyAlias,
+formatted driver name, consignment references, site names etc. The =LIKE= operator
+scans table indexes and is expensive.
+
+By adding a Couchbase caching layer, fewer requests reach SQL Server.
+
+*Scope:* FreeText Search redirection only — NOT Saved Search.
+FreeText accounts for *85–90%* of all SEB search requests.
+
+** What is Free Text vs Saved Search?
+- *Free Text Search* — user types in the Search box AND selects "All Journeys"
+- *Saved Search* — user selects any option other than "All Journeys" from the
+  "Search by preset" dropdown
+
+* Design
+
+Journey details are pushed to Couchbase when journeys are created or modified
+in the TMC database, via the Outbox Processor Service → Kafka → Couchbase.
+
+When a WebPortal user searches, the request goes to the multi-tenanted
+=SEBSearchWebApi= (deployed on OpenShift), which returns matching Journey IDs.
+Those IDs are passed back to the TMC Database for visibility/config filtering,
+then journey data is retrieved and displayed.
+
+* Release
+
+- *Available from:* 25.8
+- *Feature Flag:* ~SEB Search Improvement~
+
+** Feature Flag Behaviour
+- Automatically enabled for existing customers by the Outbox Processor Service
+  after existing journey cache is built (CERT, UAT, LIVE).
+- For *new customers* using SEB: flag must be enabled *before* creating any
+  journeys.
+
+* Rollout Instructions (TechOps)
+
+** Pre-requisites
+1. TMC system upgraded to *25.8*
+2. ~SSO.Alchemy.ApiKey~ updated with the IDAM API Key
+3. ~SEBSearchWebApi.ApiUrl~ updated with the correct SebSearchWebAPI URL
+
+** Steps
+The Outbox Processor Windows service is *disabled by default*.
+Enable it for the customer, preferably between *1:00 AM – 4:00 AM*.
+
+** SEBSearchWebAPI URLs
+| Environment | URL                                                       |
+|-------------+-----------------------------------------------------------|
+| DEV         | http://seb-search-web-api-dev.apps.gen.ocp.mms.local/    |
+| CERT        | http://seb-search-web-api-cert.apps.gen.ocp.mms.local/   |
+| UAT         | http://seb-search-web-api-uat.apps.gen.ocp.mms.local/    |
+| PROD        | http://seb-search-web-api-prod.apps.prod.ocp.mms.local/  |
+
+* UAT Refresh Instructions (DBA)
+
+Each time a TMC database is restored from PROD to UAT, run the following as
+part of the UAT refresh process. *Do NOT use TRUNCATE.*
+
+#+BEGIN_SRC sql
+DELETE FROM dataSync.tbl_SebSearchCachePublishedJourneys
+#+END_SRC
+
+This table records journeys cached to Couchbase. Deleting it causes the Outbox
+Processor to rebuild the cache for the UAT environment.
+
+* New Components
+
+** Outbox Processor Service
+- Reads from ~dataSync.tbl_OutboxEvent~ in the TMC database
+- Sends Journey metadata to a Kafka topic
+- Polls every *30 minutes* (default) for unpublished journeys
+- Deployed on: TMC Application Server
+
+** Kafka Connector (Sink)
+- Reads from Kafka topic, writes to Couchbase
+- Multi-tenant
+- Pipeline name: ~env_niot_journey_sebsearch_couchbase~
+- Very fast — consumed 1M messages in 20–25 minutes in tests
+
+** SEBSearchWebApi
+- Serves journey search requests from WebPortal
+- Returns list of Journey IDs
+- Secured via IDAM API keys
+- Deployed on OpenShift as multi-tenant service
+- Capable of automatic horizontal scaling
+
+* Journey Metadata Cached
+
+The following fields are cached (same as existing FreeText search queries):
+
+#+BEGIN_SRC 
+tbl_Journeys.RouteID
+tbl_Journeys.JourneyAlias
+fn_GetConcatResourceNameWithFormattedDriver
+  └─ tbl_Drivers.DisplayName / FullName (depending on DriverNameFormat config)
+fn_GetConcatConsignmentReferences
+  └─ tbl_ConsignmentHeader.OrderRef
+  └─ tbl_ConsignmentHeader.CustomerOrderRef
+  └─ tbl_ConsignmentHeader.DocumentID
+  └─ tbl_ConsignmentHeader.ConHeaderInfo1–5
+tbl_JourneyDrops.DropPointID
+tbl_DropPoints.DropName
+tbl_SiteType.SiteTypeName
+tbl_Journeys.StartTime
+tbl_Journeys.EndTime
+#+END_SRC
+
+* Performance Data
+
+** Journey Counts (as of 17 June 2025)
+- Total journeys across estate (up to archiving period): *8.7 million*
+- Top 5 customers by journey count (descending):
+  1. TMC_EUROCARPARTSSTORES_L
+  2. TMC_ASDA_GHS
+  3. TMC_ALLIANCEHEALTHCARE_L
+  4. TMC_GSFCARPARTS_L
+  5. TMC_TESCOUK_L
+- ASDA and Tesco UK have shorter retention periods than the other top-5
+
+** SEB Search Hit Analysis (as of 18 June 2025)
+- *5.5 million* SEB search hits in the prior month (excl. SEB Refresh hits)
+- Weekdays > weekends; *Thursday* is peak day
+- Peak working hours: *6 AM – 6 PM*
+- Peak load (9 AM – 2 PM): ~15,000 searches/hour across the estate
+- ECPS is the biggest contributor during peak hours
+- Across the full day: Sainsbury's highest, then Tesco UK
+- Max observed: *317 search requests in a single minute* (Thursday)
+
+** Test Results
+| Component            | Result                                                     |
+|----------------------+------------------------------------------------------------|
+| Outbox Processor     | 1M journeys pushed in 3–4 hours                            |
+| Kafka Sink Connector | 1M messages consumed in 20–25 minutes (even after backlog) |
+| SEBSearchWebApi      | 1,863 req/min capacity vs 438 req/min current load         |
+
+* Component Stack (all must be running)
+
+1. OutboxProcessorService
+2. TMC Database
+3. Couchbase
+4. Kafka
+5. Sink Connector
+6. IDAM API Key Authorisation Service (AKAS)
+7. OpenShift cluster
+8. SEBSearchWebApi
+9. TMC WebPortal
+10. Journey groups / user visibility config
+
+* Known Existing SEB Behaviours (pre-feature)
+
+- Clicking the *first* number in "Loaded X of Y journeys" only updates that
+  number — the search result does not change.
+- Clicking the *second* number (Y) loads all Y journeys.
+- SEB auto-refresh only refreshes displayed data — new/removed journeys
+  matching the same criteria are not added/removed dynamically.
+
+* Dashboards & Pipelines
+
+- *SEBSearchWebApi dashboard* — available in monitoring tooling
+- *Sink connector pipeline:* ~env_niot_journey_sebsearch_couchbase~
+
+* [[https://microliseuk.sharepoint.com/sites/JourneyManagement/SitePages/SEB-Search-Cache.aspx?ga=1][Links (internal)]]
+- Local dev setup (Kafka, connector, Couchbase): [see internal wiki]
+- Local dev setup (OutboxProcessorService): [see internal wiki]
+- Enable SebSearchCaching on new TMC systems: [see internal wiki]
+- Troubleshooting guide: [see internal wiki]