After launch, who fixes failures and how fast?
Start with the conclusion: on the day of an incident, speed is only half about technical strength. The other half is whether three things were written down before launch — what counts as a failure, who owns each segment of the chain, and whether the promise is "how fast someone takes it" or "how fast service is usable again". Once those three are on paper, nobody has to open a meeting to argue about who moves first. Without them, even a strong team spends the first two hours aligning perceptions.
That judgment comes with conditions: if a system is used by a dozen internal people and can be down for half a day, heavy monitoring and 24/7 on-call duty are not worth it — on-demand response is cheaper. But once it faces external customers, touches payments, or holds business records that must be retained, "who moves first and how fast an answer comes" has to be settled before launch.
Below we follow the order we actually use in custom software, mini-program / App, and back-office delivery: classify the failure first, then locate which segment of the chain it falls into, and only then discuss response targets, responsibility boundaries, and what to prepare before launch.
1. Why problems after launch so easily become disputes
There are usually three gaps between the incident scene and the contract text.
Gap one: the contract says "the system"; what breaks is a chain. One tap to place an order passes through the phone OS, the WeChat or payment platform API, your DNS and certificate, the cloud server, the database, our code, and the campaign rules your colleagues configured in the back office. Whatever segment fails, the user sees the same sentence: "the system is broken." And each segment may belong to a different company.
Gap two: "responsible for maintenance" carries no information. It does not say how fast, during which hours, or whether there is a workaround when the fix takes long. On the day of an incident, two parties can read the same sentence a day apart.
Gap three: without observability, the discussion is guesswork. No logs, no monitoring — the team on site can only reproduce first and investigate second, spending all their time confirming the symptom instead of fixing it.
Left unsolved, these three gaps produce a counterintuitive result: the failure itself may affect twenty minutes, while assigning responsibility and communicating takes two days. Real operational capability is not "how fast you fix", it is "how fast you can tell whether it is yours or someone else's".
2. Step one: it is not one failure but four
"System crashed" in user language splits into at least four types. The order of handling differs completely — especially whether the first action should be a rollback.
| Failure type | Typical symptom | Evidence | Usual first action |
|---|---|---|---|
| Availability outage | Pages fail to load, all APIs time out, blank mini-program | Error-rate spike, failed health checks, certificate or DNS anomaly | Stop the bleeding: failover, rate limiting, or roll back the previous release |
| Data error | Wrong amounts, order status stuck, inventory does not match physical stock | Normal order volume but wrong results; reproducible by order number | Freeze writes for the affected data, then decide the correction plan |
| Performance degradation | 1 second normally, 8 seconds now, queueing at peak | Response-time curve climbing, slow database queries, queue backlog | Locate whether the bottleneck is the application or the database |
| External dependency and rule changes | No payment callback, no SMS delivery, platform API upgrade errors, app delisted | Only one external link fails, and fails completely | Check the provider's announcement and credential expiry |
Among the four, data errors are the most often underestimated. The "system" opens fine, users can log in and click buttons, so teams tend not to treat it as serious at first — yet the damage is continuous: wrong data keeps being referenced, exported, and used for reconciliation. Our habit: once an incorrect write is confirmed, freeze the related write paths first. Stopping business for ten minutes is better than letting a thousand more wrong records accumulate.
The other easily missed one is external rule changes. An SMS signature rejected in review, an expired payment certificate, a retired platform API version, a category qualification expiring into a rectification request — inside the system everything is normal, not a line of code changed, but the business has stopped. Testing cannot catch these; only expiry reminders and provider announcements can.
3. Step two: which segment of the chain the failure sits in
Diagnosis depends on evidence, not statements. If no one agreed before launch what information to collect, on the day of the incident you cannot even judge "is it us". We think at least five questions must be answerable: when it started, which users and features are affected, what the error codes are, what the most recent change was, and whether a request can be traced to logs by order number.
With those five answers, responsibility usually lands in one of five segments.
One: your side's configuration and accounts. DNS records, HTTPS certificate renewal, corporate entity accounts, payment merchant status, SMS signatures, internal network allowlists, campaign rules configured in the back office. This segment fails more often than many expect, and usually when "nothing was changed" — a certificate expires, an account runs out of balance, a colleague edits a setting, all silently.
Two: the code and deployment we delivered. Defects in a new release, a miswritten configuration, a failed database migration script, dependency version conflicts. The key evidence here is "the most recent change": if the failure timing overlaps closely with a release, a config edit, or a script run, investigate in that direction first, not from the network layer.
Three: cloud and infrastructure. Provider availability-zone issues, saturated bandwidth, full disks, API abuse or attacks. On monitoring these usually show up as "overall" metric anomalies, rather than a single business path failing alone.
Four: third-party platforms and channels. Payment, SMS, maps, e-invoices, logistics, WeChat / Alipay interface adjustments, app store review policy. The signature of this type is total failure unrelated to your own version; the most effective action is to check the provider's status page and announcements, and to prepare what to tell users.
Five: platform review and compliance. An app delisted, a category asked for additional qualifications, a privacy policy that does not match actual collection sent back for rectification. Not a technical failure, but the user experiences the same "it doesn't work", and the handling cycle is measured in days.
The point of splitting responsibility is not to decide who is right, but to decide who moves first. On one chain, only one party should be acting while the others supply evidence — then incidents get fixed fast. Multiple parties changing, restarting, and guessing at the same time is the most common way to make things worse.
4. Step three: response time and restoration time are two different things
Many service commitments are vague because these two are merged into one phrase, "handled promptly".
- Response time: from the report to someone accepting it and giving an initial assessment. It promises "someone is on it".
- Restoration time: from the report to the business being usable again. It promises "business is back".
- Fix time: from root cause to a permanent fix. It promises "it will not recur".
The difference is practical: one incident can easily have a 15-minute response, a 40-minute restoration by rollback, and a proper fix three days later. If the contract only says "resolved within 24 hours", nobody dares to use the fastest tool — rollback — because rollback is not "resolved".
Severity levels are what make commitments executable. Below is how we define levels with clients in proposals; the specific numbers are a bilateral agreement, not an industry standard:
| Level | Definition | Required action |
|---|---|---|
| P1 | Core business unavailable, or data continuously wrong | Immediate engagement, stop the bleeding first, updates measured in hours |
| P2 | Main flow usable but clearly blocked or clearly slow | Engage the same day, provide a workaround |
| P3 | Local feature anomaly, single user affected | Schedule into the next release |
| P4 | Experience and optimization items | Collect, evaluate, then decide |
What actually makes an SLA usable is whether these elements are written down:
- The measurement basis for availability: which metric (API success rate or core function availability), over which period, and based on whose monitoring.
- Exclusions: client-side changes, third-party outages, and force majeure not counted — but exclusion is not the same as no help; state explicitly that "we still help diagnose, it is simply not counted against the target".
- On-call arrangement and contact channel: phone or ticket, which hours, holidays, and who is allowed to report.
- Escalation path: if unresolved past the agreed time, who joins and when. This reveals real service capability better than response time.
- Workarounds: degradation, rate limiting, switching to a backup channel, manual fallback — these usually restore business earlier than a complete fix.
- Post-incident review: which levels require one, and what it contains (timeline, root cause, impact, improvement items with owners).
A commitment that only says "24/7 on call" is worth almost nothing: it does not say how to reach a person, nor how soon after reaching them you get a conclusion.
5. Six things to prepare before launch
Calm on the day of an incident is mostly prepared in advance. Ordered by return on effort, the first four are worth doing even on a tight budget.
1. Monitoring and alerts on key metrics. Full-stack observability is not needed on day one. Watch four things: error rate on core APIs, business success rate (orders placed, payment callbacks, messages delivered), response time, and queue backlog with failed jobs. Alerts must name recipients and channels — an alert sent into a group nobody reads equals no alert.
2. Traceable logs. Every request carries a unique request ID so that an order number or phone number a user reports can be traced to the actual call chain. Retention must match the business reconciliation cycle; minimize personal-data fields and do not dump sensitive data to disk "to make debugging easier".
3. On-call roster and contact list. Each side names at least two contacts with hours and escalation order. A single contact on leave, gone, or with a changed number wastes the first hour of an incident on phone calls.
4. Rollback plan. The previous release must remain rollback-ready: images and configuration retained, database changes either reversible or forward-compatible. Mini-programs need attention to version state after review and release and how to revert; apps must keep API compatibility for older versions, so a server change does not break every user on an old build.
5. Backup and restore drills. A backup existing does not mean it can be restored; a successful scheduled job does not mean the data is usable. Run one full drill before launch and record the actual restore time — that number is your real confidence on a bad day.
6. Account and permission handover. Cloud accounts, domains and certificates, merchant IDs, SMS signatures, app store developer accounts, third-party sub-accounts: your company should hold entity ownership, and we receive operating permissions as needed. Hand over an account list marked with ownership, renewal dates, and responsible people. This is also the key to reducing vendor dependency: with accounts in your hands, changing any provider does not stop the business.
6. The first hour after an incident, in this order
Wrong order wastes time no matter how strong the engineering.
- Confirm the impact first: which users, which features, is there an alternative path (if the mini-program is down, does the app still work). This decides whether you speak publicly, and when.
- Stop the bleeding before hunting the root cause: rollback, degradation, rate limiting, switching channels, manual fallback. Users care whether it works, not why.
- Keep one spokesperson per side: one on the client side, one on the service side, on a fixed cadence (say every 30 minutes), so information does not contradict itself across groups.
- Preserve the scene: logs, error screenshots, request samples, the configuration and version in use — save before restarting or cleaning up. Many root causes are erased by "just restart it".
- Keep a timeline: what time it was noticed, who did what, when it recovered. Reviews and responsibility split depend on this record; do not rely on memory.
- Conclusion the same day, review within three days: state impact and temporary handling the same day; put root cause and improvements in the review, with owners and dates.
7. Different teams should choose differently
The same method has to be adjusted per company.
Teams with no technical staff and no plan to hire. Delegate operations as a whole, but write service hours, response terms, and monthly inspections into the contract, and require a monthly report you can actually read: availability numbers, incident count, handling durations, open improvement items. You do not need to understand engineering, but you need to be able to judge whether the service was delivered this month.
Teams with their own IT. The biggest waste is overlapping ownership: both sides think the other is handling it, or both act at once. Build a chain responsibility table — segment by segment, who runs daily operations, who makes changes, who moves first during an incident — and run one joint drill before launch. One drill costs far less than the confusion of a first real incident.
Systems assembled from multiple vendors (front end, back end, implementation). This is the structure most prone to disputes. Either appoint one coordinator accountable for the whole, or at minimum keep a chain responsibility table and a single reporting entry point. Without it, your first hour is spent watching three vendors present evidence at each other.
Teams evaluating a new system. Ask three questions during requirements: who is on call after launch, how severities are defined, and do you have review samples that can be verified. The answers say more about delivery quality than any brochure.
8. How Boiling Water Technology handles this, and where our boundaries are
We write it into deliverables rather than promises:
- At proposal stage, confirm with the client the severity levels, response and restoration terms, contacts on both sides and the escalation path, as a contract annex;
- Before launch, produce a go-live checklist covering monitored metrics, alert recipients, rollback plan, backup and restore drill records, and the account ownership list;
- At handover, transfer source code, deployment documentation and the account list together; the client holds the primary accounts and we keep the necessary operating permissions;
- After launch, a named contact handles reports through a single entry point; P1 incidents get same-day updates on impact and temporary handling, followed by a review and improvement items.
Boundaries we should state plainly: third-party platform capabilities, review outcomes and rule changes are not ours to decide — what we can do is warn early, propose alternatives, and help communicate. Your entity qualifications, accounts, payment merchant IDs and internal network are held and authorized by you; we help diagnose but cannot hold or authorize them on your behalf. New requirements outside the agreed scope are changes and go through change assessment, not incident handling.
Conversely, we also advise some clients not to buy heavy operations: if a system serves a few internal roles and half a day of downtime does not affect external business or money, per-incident support plus quarterly inspection is enough, and the budget is better spent on feature iteration.
9. Additional questions
A provider says it is "on call 24/7". Is that enough?
No. Ask three follow-ups: how do we reach a person (phone or ticket, is there a roster), how soon is an initial assessment, and after how long does someone escalate. Without those three, 24/7 is an adjective.
If the incident is caused by WeChat or the payment platform, whose responsibility is it?
Usually outside the service target, but helping diagnose, switching to a backup channel, and preparing what to tell users should still be in scope. Write "excluded from targets" and "still assisted" separately in the contract, so nobody ends the conversation with "not our problem".
We have no technical staff — how do we verify that operations are well done?
Look at three things: availability data and incident records delivered monthly, a review after every P1, and whether the improvement items in those reviews were actually completed. All three are verifiable without a technical background.
Can new features after launch be counted as incident handling?
Better not to mix them. An incident is "a promised function not working as expected"; a new requirement is a scope change. Scheduling, pricing, and acceptance differ. Separating the processes keeps both sides from accumulating resentment.
Sources
The judgments in this article come from our delivery, launch, and operations practice in custom software, mini-program / App and back-office system projects, together with general engineering practice on incident severity and availability measurement. The severity scheme, timing elements, and readiness checklist are our recommended approach; the specific numbers must be agreed by both parties based on business impact and are not an industry standard. Platform capabilities, review rules, and qualification requirements follow each platform's official documentation as of the day you read it; no specific vendor clause numbers are cited here.