178 lines
6.7 KiB
Org Mode
Executable File
178 lines
6.7 KiB
Org Mode
Executable File
:PROPERTIES:
|
||
:ID: 0b6e7dc0-1171-4dc4-982e-a5d95765e09c
|
||
:END:
|
||
#+TITLE: SEB Search Improvement — Couchbase Caching
|
||
#+filetags: :couchbase:kafka:
|
||
#+DATE: 2025-06-18
|
||
|
||
* Overview
|
||
|
||
DBAs are seeing high CPU and memory usage on SQL production servers due to SEB
|
||
Search. Free text search uses the =LIKE= operator against RouteID, JourneyAlias,
|
||
formatted driver name, consignment references, site names etc. The =LIKE= operator
|
||
scans table indexes and is expensive.
|
||
|
||
By adding a Couchbase caching layer, fewer requests reach SQL Server.
|
||
|
||
*Scope:* FreeText Search redirection only — NOT Saved Search.
|
||
FreeText accounts for *85–90%* of all SEB search requests.
|
||
|
||
** What is Free Text vs Saved Search?
|
||
- *Free Text Search* — user types in the Search box AND selects "All Journeys"
|
||
- *Saved Search* — user selects any option other than "All Journeys" from the
|
||
"Search by preset" dropdown
|
||
|
||
* Design
|
||
|
||
Journey details are pushed to Couchbase when journeys are created or modified
|
||
in the TMC database, via the Outbox Processor Service → Kafka → Couchbase.
|
||
|
||
When a WebPortal user searches, the request goes to the multi-tenanted
|
||
=SEBSearchWebApi= (deployed on OpenShift), which returns matching Journey IDs.
|
||
Those IDs are passed back to the TMC Database for visibility/config filtering,
|
||
then journey data is retrieved and displayed.
|
||
|
||
* Release
|
||
|
||
- *Available from:* 25.8
|
||
- *Feature Flag:* ~SEB Search Improvement~
|
||
|
||
** Feature Flag Behaviour
|
||
- Automatically enabled for existing customers by the Outbox Processor Service
|
||
after existing journey cache is built (CERT, UAT, LIVE).
|
||
- For *new customers* using SEB: flag must be enabled *before* creating any
|
||
journeys.
|
||
|
||
* Rollout Instructions (TechOps)
|
||
|
||
** Pre-requisites
|
||
1. TMC system upgraded to *25.8*
|
||
2. ~SSO.Alchemy.ApiKey~ updated with the IDAM API Key
|
||
3. ~SEBSearchWebApi.ApiUrl~ updated with the correct SebSearchWebAPI URL
|
||
|
||
** Steps
|
||
The Outbox Processor Windows service is *disabled by default*.
|
||
Enable it for the customer, preferably between *1:00 AM – 4:00 AM*.
|
||
|
||
** SEBSearchWebAPI URLs
|
||
| Environment | URL |
|
||
|-------------+-----------------------------------------------------------|
|
||
| DEV | http://seb-search-web-api-dev.apps.gen.ocp.mms.local/ |
|
||
| CERT | http://seb-search-web-api-cert.apps.gen.ocp.mms.local/ |
|
||
| UAT | http://seb-search-web-api-uat.apps.gen.ocp.mms.local/ |
|
||
| PROD | http://seb-search-web-api-prod.apps.prod.ocp.mms.local/ |
|
||
|
||
* UAT Refresh Instructions (DBA)
|
||
|
||
Each time a TMC database is restored from PROD to UAT, run the following as
|
||
part of the UAT refresh process. *Do NOT use TRUNCATE.*
|
||
|
||
#+BEGIN_SRC sql
|
||
DELETE FROM dataSync.tbl_SebSearchCachePublishedJourneys
|
||
#+END_SRC
|
||
|
||
This table records journeys cached to Couchbase. Deleting it causes the Outbox
|
||
Processor to rebuild the cache for the UAT environment.
|
||
|
||
* New Components
|
||
|
||
** Outbox Processor Service
|
||
- Reads from ~dataSync.tbl_OutboxEvent~ in the TMC database
|
||
- Sends Journey metadata to a Kafka topic
|
||
- Polls every *30 minutes* (default) for unpublished journeys
|
||
- Deployed on: TMC Application Server
|
||
|
||
** Kafka Connector (Sink)
|
||
- Reads from Kafka topic, writes to Couchbase
|
||
- Multi-tenant
|
||
- Pipeline name: ~env_niot_journey_sebsearch_couchbase~
|
||
- Very fast — consumed 1M messages in 20–25 minutes in tests
|
||
|
||
** SEBSearchWebApi
|
||
- Serves journey search requests from WebPortal
|
||
- Returns list of Journey IDs
|
||
- Secured via IDAM API keys
|
||
- Deployed on OpenShift as multi-tenant service
|
||
- Capable of automatic horizontal scaling
|
||
|
||
* Journey Metadata Cached
|
||
|
||
The following fields are cached (same as existing FreeText search queries):
|
||
|
||
#+BEGIN_SRC
|
||
tbl_Journeys.RouteID
|
||
tbl_Journeys.JourneyAlias
|
||
fn_GetConcatResourceNameWithFormattedDriver
|
||
└─ tbl_Drivers.DisplayName / FullName (depending on DriverNameFormat config)
|
||
fn_GetConcatConsignmentReferences
|
||
└─ tbl_ConsignmentHeader.OrderRef
|
||
└─ tbl_ConsignmentHeader.CustomerOrderRef
|
||
└─ tbl_ConsignmentHeader.DocumentID
|
||
└─ tbl_ConsignmentHeader.ConHeaderInfo1–5
|
||
tbl_JourneyDrops.DropPointID
|
||
tbl_DropPoints.DropName
|
||
tbl_SiteType.SiteTypeName
|
||
tbl_Journeys.StartTime
|
||
tbl_Journeys.EndTime
|
||
#+END_SRC
|
||
|
||
* Performance Data
|
||
|
||
** Journey Counts (as of 17 June 2025)
|
||
- Total journeys across estate (up to archiving period): *8.7 million*
|
||
- Top 5 customers by journey count (descending):
|
||
1. TMC_EUROCARPARTSSTORES_L
|
||
2. TMC_ASDA_GHS
|
||
3. TMC_ALLIANCEHEALTHCARE_L
|
||
4. TMC_GSFCARPARTS_L
|
||
5. TMC_TESCOUK_L
|
||
- ASDA and Tesco UK have shorter retention periods than the other top-5
|
||
|
||
** SEB Search Hit Analysis (as of 18 June 2025)
|
||
- *5.5 million* SEB search hits in the prior month (excl. SEB Refresh hits)
|
||
- Weekdays > weekends; *Thursday* is peak day
|
||
- Peak working hours: *6 AM – 6 PM*
|
||
- Peak load (9 AM – 2 PM): ~15,000 searches/hour across the estate
|
||
- ECPS is the biggest contributor during peak hours
|
||
- Across the full day: Sainsbury's highest, then Tesco UK
|
||
- Max observed: *317 search requests in a single minute* (Thursday)
|
||
|
||
** Test Results
|
||
| Component | Result |
|
||
|----------------------+------------------------------------------------------------|
|
||
| Outbox Processor | 1M journeys pushed in 3–4 hours |
|
||
| Kafka Sink Connector | 1M messages consumed in 20–25 minutes (even after backlog) |
|
||
| SEBSearchWebApi | 1,863 req/min capacity vs 438 req/min current load |
|
||
|
||
* Component Stack (all must be running)
|
||
|
||
1. OutboxProcessorService
|
||
2. TMC Database
|
||
3. Couchbase
|
||
4. Kafka
|
||
5. Sink Connector
|
||
6. IDAM API Key Authorisation Service (AKAS)
|
||
7. OpenShift cluster
|
||
8. SEBSearchWebApi
|
||
9. TMC WebPortal
|
||
10. Journey groups / user visibility config
|
||
|
||
* Known Existing SEB Behaviours (pre-feature)
|
||
|
||
- Clicking the *first* number in "Loaded X of Y journeys" only updates that
|
||
number — the search result does not change.
|
||
- Clicking the *second* number (Y) loads all Y journeys.
|
||
- SEB auto-refresh only refreshes displayed data — new/removed journeys
|
||
matching the same criteria are not added/removed dynamically.
|
||
|
||
* Dashboards & Pipelines
|
||
|
||
- *SEBSearchWebApi dashboard* — available in monitoring tooling
|
||
- *Sink connector pipeline:* ~env_niot_journey_sebsearch_couchbase~
|
||
|
||
* [[https://microliseuk.sharepoint.com/sites/JourneyManagement/SitePages/SEB-Search-Cache.aspx?ga=1][Links (internal)]]
|
||
- Local dev setup (Kafka, connector, Couchbase): [see internal wiki]
|
||
- Local dev setup (OutboxProcessorService): [see internal wiki]
|
||
- Enable SebSearchCaching on new TMC systems: [see internal wiki]
|
||
- Troubleshooting guide: [see internal wiki]
|