adding content

Career Concepts/20260410122425-ess_database.org

@@ -0,0 +1,127 @@
+:PROPERTIES:
+:ID:       46add22d-e562-4e3a-a301-d4aea2552952
+:END:
+#+DATE: 2026-04-10
+#+filetags: :database:sql:ess:esp:microlise:
+#+STARTUP: showall
+#+title: ESP — Database & Dacpac
+
+* The ESP Database
+:PROPERTIES:
+:RELATED: [[id:a6c345df-8db9-4538-b87e-0e72e2414905][ESP Deployment Pipeline — Index]] 
+:END:
+
+ESP uses a single SQL Server database called *EspBroker*. This is deployed and upgraded as part of the overall ESP deployment process.
+
+* What Is a Dacpac?
+
+A *dacpac* (Data-tier Application Package) is a packaged representation of a SQL Server database *schema*. It is a file with the extension ~.dacpac~.
+
+Rather than writing migration scripts that say "ALTER TABLE, ADD COLUMN..." manually, you instead describe the *desired end state* of the database, and the dacpac deployment tool (~sqlpackage~) calculates the delta and applies it automatically.
+
+** How Dacpac Deployment Works
+#+BEGIN_SRC
+Your dacpac file (desired schema)
+         │
+         ▼
+    sqlpackage.exe
+         │
+         ├── Connects to target database
+         ├── Compares desired schema to actual schema
+         ├── Generates a diff
+         └── Applies the diff (ALTER TABLE, CREATE INDEX, etc.)
+#+END_SRC
+
+** Advantages Over Raw SQL Scripts
+| Dacpac                                  | Raw SQL Scripts                           |
+|-----------------------------------------+-------------------------------------------|
+| Declarative — describe what you want    | Imperative — describe each change step    |
+| Tool calculates the diff automatically  | You must track and apply changes manually |
+| Idempotent — safe to run multiple times | Can fail if run twice without care        |
+| Schema is versioned as code             | Scripts can get out of sync               |
+
+** Relevant PowerShell / CLI
+#+BEGIN_SRC powershell
+# Deploy a dacpac using sqlpackage
+& "C:\Program Files\Microsoft SQL Server\160\DAC\bin\sqlpackage.exe" `
+    /Action:Publish `
+    /SourceFile:"EspBroker.dacpac" `
+    /TargetServerName:"ROMAC-DEV-DB01" `
+    /TargetDatabaseName:"EspBroker"
+#+END_SRC
+
+* Team Ludo's Dacpac
+
+Team Ludo built a dacpac solution for EspBroker. This is already in the repo and represents the *target schema* that all customer databases should eventually conform to.
+
+* The Critical Problem — Schema Inconsistency
+
+** What the Problem Is
+Existing customer databases have drifted from the official schema over time. This drift likely happened due to:
+- Manual hotfixes applied directly to production databases
+- Different versions of ESP deployed to different customers at different times
+- No enforced schema management historically
+
+This means the dacpac's expected schema does not match what is actually in customer databases.
+
+** Consequence
+If you attempt to deploy the dacpac against an inconsistent database, ~sqlpackage~ will either:
+- Fail with errors (best case — nothing is changed)
+- Apply incorrect changes that corrupt data (worst case)
+
+** What Needs to Happen
+Before the dacpac can be used for a customer:
+1. A database expert must *manually inspect* the customer's database
+2. Identify all schema differences between actual and expected state
+3. Write and apply *manual SQL scripts* to bring the DB into alignment
+4. Verify the dacpac can then deploy cleanly (ideally against a copy)
+5. Only then mark the customer as ~dacpacReady: true~ in the manifest
+
+** Pipeline Safeguard
+The deployment pipeline *must* check the ~dacpacReady~ flag before attempting database deployment, and fail clearly if it is ~false~:
+
+#+BEGIN_SRC powershell
+function Invoke-DatabaseDeployment {
+    param($Manifest)
+
+    if (-not $Manifest.Database.DacpacReady) {
+        Write-Error "Database for $($Manifest.Customer) $($Manifest.Environment) is not dacpac-ready. " +
+                    "Manual schema alignment is required before deployment."
+        throw "Database not ready for dacpac deployment."
+    }
+
+    # Proceed with dacpac deployment...
+    Deploy-Dacpac -DacpacPath $artifactPath -Server $Manifest.Servers.DbServer -Database "EspBroker"
+}
+#+END_SRC
+
+* Database Deployment in the Deployment Stage
+
+Database deployment sits inside the *DEPLOY stage*. See [[id:cd3cd02f-c9b3-4455-8ce5-b2a5aa458fed][ESP — Deployment Stages]] for full stage detail. The database is typically deployed before services are started up to ensure the schema is ready for the new application code.
+
+Suggested order within the DEPLOY stage:
+1. Stop Windows Services
+2. Deploy database dacpac (if dacpacReady)
+3. Handle Citrix sessions
+4. Swap application files
+5. Start Windows Services back up
+
+* Rollback Considerations
+
+If the dacpac deploys successfully but a subsequent step fails, rolling back the database is non-trivial. Dacpac does not natively support rollback — options are:
+- Restore from backup (requires a backup to have been taken immediately before)
+- Write a counter-dacpac that reverts the schema (complex, error-prone)
+
+This is one of the reasons the *rollback plan is still TBC*. See [[id:82c3d447-d6d3-498e-9f66-aee60c752462][ESP — Open Questions & TBC Items]].
+
+A *database backup must be taken before any deployment* — this should be a mandatory step in the PREDEPLOY stage.
+
+* SQL Server Concepts Relevant Here
+
+| Concept        | Relevance                                                    |
+|----------------+--------------------------------------------------------------|
+| Schema         | The structure of tables, columns, indexes, constraints, etc. |
+| ~sqlpackage~   | Microsoft CLI tool that applies dacpac files                 |
+| ~BACPAC~       | Like dacpac but includes data — useful for backup/restore    |
+| SQL Agent Jobs | Scheduled SQL jobs that may need to be handled during deploy |
+| Linked Servers | DB connections to other servers — may be part of ESP's setup |