org_roam/Career Concepts/20260505121135-seb_search_improvements.org

SEB Search Improvement — Couchbase Caching

• Overview
  • What is Free Text vs Saved Search?
• Design
• Release
  • Feature Flag Behaviour
• Rollout Instructions (TechOps)
  • Pre-requisites
  • Steps
  • SEBSearchWebAPI URLs
• UAT Refresh Instructions (DBA)
• New Components
  • Outbox Processor Service
  • Kafka Connector (Sink)
  • SEBSearchWebApi
• Journey Metadata Cached
• Performance Data
  • Journey Counts (as of 17 June 2025)
  • SEB Search Hit Analysis (as of 18 June 2025)
  • Test Results
• Component Stack (all must be running)
• Known Existing SEB Behaviours (pre-feature)
• Dashboards & Pipelines
• Links (internal)

Overview

DBAs are seeing high CPU and memory usage on SQL production servers due to SEB Search. Free text search uses the LIKE operator against RouteID, JourneyAlias, formatted driver name, consignment references, site names etc. The LIKE operator scans table indexes and is expensive.

By adding a Couchbase caching layer, fewer requests reach SQL Server.

Scope: FreeText Search redirection only — NOT Saved Search. FreeText accounts for 85–90% of all SEB search requests.

What is Free Text vs Saved Search?

• Free Text Search — user types in the Search box AND selects "All Journeys"
• Saved Search — user selects any option other than "All Journeys" from the "Search by preset" dropdown

Design

Journey details are pushed to Couchbase when journeys are created or modified in the TMC database, via the Outbox Processor Service → Kafka → Couchbase.

When a WebPortal user searches, the request goes to the multi-tenanted SEBSearchWebApi (deployed on OpenShift), which returns matching Journey IDs. Those IDs are passed back to the TMC Database for visibility/config filtering, then journey data is retrieved and displayed.

Release

• Available from: 25.8
• Feature Flag: SEB Search Improvement

Feature Flag Behaviour

• Automatically enabled for existing customers by the Outbox Processor Service after existing journey cache is built (CERT, UAT, LIVE).
• For new customers using SEB: flag must be enabled before creating any journeys.

Rollout Instructions (TechOps)

Pre-requisites

1. TMC system upgraded to 25.8
2. SSO.Alchemy.ApiKey updated with the IDAM API Key
3. SEBSearchWebApi.ApiUrl updated with the correct SebSearchWebAPI URL

Steps

The Outbox Processor Windows service is disabled by default. Enable it for the customer, preferably between 1:00 AM – 4:00 AM.

SEBSearchWebAPI URLs

Environment	URL
DEV	http://seb-search-web-api-dev.apps.gen.ocp.mms.local/
CERT	http://seb-search-web-api-cert.apps.gen.ocp.mms.local/
UAT	http://seb-search-web-api-uat.apps.gen.ocp.mms.local/
PROD	http://seb-search-web-api-prod.apps.prod.ocp.mms.local/

UAT Refresh Instructions (DBA)

Each time a TMC database is restored from PROD to UAT, run the following as part of the UAT refresh process. Do NOT use TRUNCATE.

DELETE FROM dataSync.tbl_SebSearchCachePublishedJourneys

This table records journeys cached to Couchbase. Deleting it causes the Outbox Processor to rebuild the cache for the UAT environment.

New Components

Outbox Processor Service

• Reads from dataSync.tbl_OutboxEvent in the TMC database
• Sends Journey metadata to a Kafka topic
• Polls every 30 minutes (default) for unpublished journeys
• Deployed on: TMC Application Server

Kafka Connector (Sink)

• Reads from Kafka topic, writes to Couchbase
• Multi-tenant
• Pipeline name: env_niot_journey_sebsearch_couchbase
• Very fast — consumed 1M messages in 20–25 minutes in tests

SEBSearchWebApi

• Serves journey search requests from WebPortal
• Returns list of Journey IDs
• Secured via IDAM API keys
• Deployed on OpenShift as multi-tenant service
• Capable of automatic horizontal scaling

Journey Metadata Cached

The following fields are cached (same as existing FreeText search queries):

tbl_Journeys.RouteID
tbl_Journeys.JourneyAlias
fn_GetConcatResourceNameWithFormattedDriver
  └─ tbl_Drivers.DisplayName / FullName (depending on DriverNameFormat config)
fn_GetConcatConsignmentReferences
  └─ tbl_ConsignmentHeader.OrderRef
  └─ tbl_ConsignmentHeader.CustomerOrderRef
  └─ tbl_ConsignmentHeader.DocumentID
  └─ tbl_ConsignmentHeader.ConHeaderInfo1–5
tbl_JourneyDrops.DropPointID
tbl_DropPoints.DropName
tbl_SiteType.SiteTypeName
tbl_Journeys.StartTime
tbl_Journeys.EndTime

Performance Data

Journey Counts (as of 17 June 2025)

• Total journeys across estate (up to archiving period): 8.7 million

• Top 5 customers by journey count (descending):

  1. TMC_EUROCARPARTSSTORES_L
  2. TMC_ASDA_GHS
  3. TMC_ALLIANCEHEALTHCARE_L
  4. TMC_GSFCARPARTS_L
  5. TMC_TESCOUK_L
• ASDA and Tesco UK have shorter retention periods than the other top-5

SEB Search Hit Analysis (as of 18 June 2025)

• 5.5 million SEB search hits in the prior month (excl. SEB Refresh hits)
• Weekdays > weekends; Thursday is peak day
• Peak working hours: 6 AM – 6 PM
• Peak load (9 AM – 2 PM): ~15,000 searches/hour across the estate
• ECPS is the biggest contributor during peak hours
• Across the full day: Sainsbury's highest, then Tesco UK
• Max observed: 317 search requests in a single minute (Thursday)

Test Results

Component	Result
Outbox Processor	1M journeys pushed in 3–4 hours
Kafka Sink Connector	1M messages consumed in 20–25 minutes (even after backlog)
SEBSearchWebApi	1,863 req/min capacity vs 438 req/min current load

Component Stack (all must be running)

1. OutboxProcessorService
2. TMC Database
3. Couchbase
4. Kafka
5. Sink Connector
6. IDAM API Key Authorisation Service (AKAS)
7. OpenShift cluster
8. SEBSearchWebApi
9. TMC WebPortal
10. Journey groups / user visibility config

Known Existing SEB Behaviours (pre-feature)

• Clicking the first number in "Loaded X of Y journeys" only updates that number — the search result does not change.
• Clicking the second number (Y) loads all Y journeys.
• SEB auto-refresh only refreshes displayed data — new/removed journeys matching the same criteria are not added/removed dynamically.

Dashboards & Pipelines

• SEBSearchWebApi dashboard — available in monitoring tooling
• Sink connector pipeline: env_niot_journey_sebsearch_couchbase

Links (internal)

• Local dev setup (Kafka, connector, Couchbase): [see internal wiki]
• Local dev setup (OutboxProcessorService): [see internal wiki]
• Enable SebSearchCaching on new TMC systems: [see internal wiki]
• Troubleshooting guide: [see internal wiki]