---
note type:
  - microlise
  - note
date: 2026-06-03
done: true
---
# Overview

DBAs are seeing high CPU and memory usage on SQL production servers due to SEB
Search. Free text search uses the `LIKE` operator against RouteID, JourneyAlias,
formatted driver name, consignment references, site names etc. The `LIKE` operator
scans table indexes and is expensive.

By adding a Couchbase caching layer, fewer requests reach SQL Server.

**Scope:** FreeText Search redirection only — NOT Saved Search.
FreeText accounts for **85–90%** of all SEB search requests.

## What is Free Text vs Saved Search?

  - **Free Text Search** — user types in the Search box AND selects "All Journeys"
  - **Saved Search** — user selects any option other than "All Journeys" from the
    "Search by preset" dropdown

# Design

Journey details are pushed to Couchbase when journeys are created or modified
in the TMC database, via the Outbox Processor Service → Kafka → Couchbase.

When a WebPortal user searches, the request goes to the multi-tenanted
`SEBSearchWebApi` (deployed on OpenShift), which returns matching Journey IDs.
Those IDs are passed back to the TMC Database for visibility/config filtering,
then journey data is retrieved and displayed.

# Release

  - **Available from:** 25.8
  - **Feature Flag:** `SEB Search Improvement`

## Feature Flag Behaviour

  - Automatically enabled for existing customers by the Outbox Processor Service
    after existing journey cache is built (CERT, UAT, LIVE).
  - For **new customers** using SEB: flag must be enabled **before** creating any
    journeys.

# Rollout Instructions (TechOps)

## Pre-requisites

1.  TMC system upgraded to **25.8**
2.  `SSO.Alchemy.ApiKey` updated with the IDAM API Key
3.  `SEBSearchWebApi.ApiUrl` updated with the correct SebSearchWebAPI URL

## Steps

The Outbox Processor Windows service is **disabled by default**.
Enable it for the customer, preferably between **1:00 AM – 4:00 AM**.

## SEBSearchWebAPI URLs

| Environment | URL                                                       |
| ----------- | --------------------------------------------------------- |
| DEV         | <http://seb-search-web-api-dev.apps.gen.ocp.mms.local/>   |
| CERT        | <http://seb-search-web-api-cert.apps.gen.ocp.mms.local/>  |
| UAT         | <http://seb-search-web-api-uat.apps.gen.ocp.mms.local/>   |
| PROD        | <http://seb-search-web-api-prod.apps.prod.ocp.mms.local/> |

# UAT Refresh Instructions (DBA)

Each time a TMC database is restored from PROD to UAT, run the following as
part of the UAT refresh process. **Do NOT use TRUNCATE.**

``` sql
DELETE FROM dataSync.tbl_SebSearchCachePublishedJourneys
```

This table records journeys cached to Couchbase. Deleting it causes the Outbox
Processor to rebuild the cache for the UAT environment.

# New Components

## Outbox Processor Service

  - Reads from `dataSync.tbl_OutboxEvent` in the TMC database
  - Sends Journey metadata to a Kafka topic
  - Polls every **30 minutes** (default) for unpublished journeys
  - Deployed on: TMC Application Server

## Kafka Connector (Sink)

  - Reads from Kafka topic, writes to Couchbase
  - Multi-tenant
  - Pipeline name: `env_niot_journey_sebsearch_couchbase`
  - Very fast — consumed 1M messages in 20–25 minutes in tests

## SEBSearchWebApi

  - Serves journey search requests from WebPortal
  - Returns list of Journey IDs
  - Secured via IDAM API keys
  - Deployed on OpenShift as multi-tenant service
  - Capable of automatic horizontal scaling

# Journey Metadata Cached

The following fields are cached (same as existing FreeText search queries):

    tbl_Journeys.RouteID
    tbl_Journeys.JourneyAlias
    fn_GetConcatResourceNameWithFormattedDriver
      └─ tbl_Drivers.DisplayName / FullName (depending on DriverNameFormat config)
    fn_GetConcatConsignmentReferences
      └─ tbl_ConsignmentHeader.OrderRef
      └─ tbl_ConsignmentHeader.CustomerOrderRef
      └─ tbl_ConsignmentHeader.DocumentID
      └─ tbl_ConsignmentHeader.ConHeaderInfo1–5
    tbl_JourneyDrops.DropPointID
    tbl_DropPoints.DropName
    tbl_SiteType.SiteTypeName
    tbl_Journeys.StartTime
    tbl_Journeys.EndTime

# Performance Data

## Journey Counts (as of 17 June 2025)

  - Total journeys across estate (up to archiving period): **8.7 million**
  - Top 5 customers by journey count (descending):
    1.  TMC<sub>EUROCARPARTSSTORESL</sub>
    2.  TMC<sub>ASDAGHS</sub>
    3.  TMC<sub>ALLIANCEHEALTHCAREL</sub>
    4.  TMC<sub>GSFCARPARTSL</sub>
    5.  TMC<sub>TESCOUKL</sub>
  - ASDA and Tesco UK have shorter retention periods than the other top-5

## SEB Search Hit Analysis (as of 18 June 2025)

  - **5.5 million** SEB search hits in the prior month (excl. SEB Refresh hits)
  - Weekdays \> weekends; **Thursday** is peak day
  - Peak working hours: **6 AM – 6 PM**
  - Peak load (9 AM – 2 PM): \~15,000 searches/hour across the estate
  - ECPS is the biggest contributor during peak hours
  - Across the full day: Sainsbury's highest, then Tesco UK
  - Max observed: **317 search requests in a single minute** (Thursday)

## Test Results

| Component            | Result                                                     |
| -------------------- | ---------------------------------------------------------- |
| Outbox Processor     | 1M journeys pushed in 3–4 hours                            |
| Kafka Sink Connector | 1M messages consumed in 20–25 minutes (even after backlog) |
| SEBSearchWebApi      | 1,863 req/min capacity vs 438 req/min current load         |

# Component Stack (all must be running)

1.  OutboxProcessorService
2.  TMC Database
3.  Couchbase
4.  Kafka
5.  Sink Connector
6.  IDAM API Key Authorisation Service (AKAS)
7.  OpenShift cluster
8.  SEBSearchWebApi
9.  TMC WebPortal
10. Journey groups / user visibility config

# Known Existing SEB Behaviours (pre-feature)

  - Clicking the **first** number in "Loaded X of Y journeys" only updates that
    number — the search result does not change.
  - Clicking the **second** number (Y) loads all Y journeys.
  - SEB auto-refresh only refreshes displayed data — new/removed journeys
    matching the same criteria are not added/removed dynamically.

# Dashboards & Pipelines

  - **SEBSearchWebApi dashboard** — available in monitoring tooling
  - **Sink connector pipeline:** `env_niot_journey_sebsearch_couchbase`

# [Links (internal)](https://microliseuk.sharepoint.com/sites/JourneyManagement/SitePages/SEB-Search-Cache.aspx?ga=1)

  - Local dev setup (Kafka, connector, Couchbase): \[see internal wiki\]
  - Local dev setup (OutboxProcessorService): \[see internal wiki\]
  - Enable SebSearchCaching on new TMC systems: \[see internal wiki\]
  - Troubleshooting guide: \[see internal wiki\]