Sprint plan

Five gaps found by running the system on 20 Aug, not by reading it. All closed and merged, each with a CI assertion that exercises the real thing, because two of them (the gateway and the boundary) had broken while every static check passed.

• /ready hung ~130 s against an unreachable database; probes now run off the event loop under a deadline and return 503 in 2 s while /health keeps answering (mib-shared#3).
• Nothing routed through the gateway: Traefik could not read the Docker socket, and CLI flags in the dev override discarded the static config, so the entrypoint every label routes on did not exist. CI now routes a real request (mib-infra#12).
• The boundary was not enforced: all six services connected as the database owner. Each now uses its own least-privilege role with passwords from the environment; CI asserts the refusal against a real cluster (mib-infra#4).
• mib-shared was consumed by nothing: now adopted everywhere, duplicated logging.py deleted; adoption surfaced a swallowed traceback and a missing trace_id (mib-shared#4).
• No tracing: W3C propagation, the error envelope with a quotable reference, and a client with a mandatory timeout, bounded retry and defined fallback (mib-shared#2).

Gate

A request through Traefik returns 200 from a service. /ready returns 503 in about two seconds with the database down while /health keeps answering. Traefik initiates the trace and every service carries trace_id on every log line. A service attempting a cross-schema read is refused with a permission error. Every clause is asserted in CI on each change.

On a single self-managed host the database holds the corpus, the vector index and every user and payment record, with nothing managed behind it, so two of this stage's gates are release gates.

Build

Egress for Qwen, OSS, the payment gateway (Midtrans) and the crawl sources on the host firewall. Inbound restricted to Cloudflare ranges plus administrative SSH. Host provisioned from version-controlled configuration. A decision on secret injection. The memory budget written down, including the resident vector index. Continuous WAL archiving to OSS with failure alerting. Development, staging and production, with production never sharing a host.

Gate

A full restore rehearsed on a clean host, with the elapsed time recorded against the four-hour target (NFR-38, NFR-39): measured, not asserted. Backup and archive failures both raise an alert. The load-test harness (NFR-35, mib-infra#7) sustains the target rate against the real topology, including a login burst and identity's peak memory.

• Built Compose projects for five environments, now one directory per service with env overlays (mib-infra#74, 13 Oct); Traefik routing; Postgres with pgvector; least-privilege roles; observability; CI hygiene checks that fail the build if anything but Traefik publishes a host port.
• 25 Aug Performance targets adopted by assumption, since the client has no figures: ≤1 s lexical search and filter (NFR-01), ≤2 s detail (NFR-02), ≤15 s AI summary (NFR-03), in mib-infra/docs/performance-targets.md (mib-infra#10). Validating them waits on the load harness and a real corpus.
• 26 Aug Rate limits adopted by assumption (mib-infra#11, PR #35): API per IP at the edge, 20 req/s average, burst 40, through a shared api-ratelimit middleware; PDF downloads per account 50/day, burst 5/minute, handed to mib-regulations as config (mib-regulations#6). Per-user limiting is unimplemented anywhere.
• 26 Aug Maintenance mode (mib-infra#21, PR #33): a Traefik file-provider catch-all that outranks every label router, scripts/maintenance.sh on/off, a mib_maintenance_active flag and alert suppression. The orchestrator pipeline-cycle.sh (mib-infra#36, PR #37) ends the window when ingestion finishes and refuses to start while embedding is still draining.
• 29 Aug Observability's topology decided (mib-infra#20, PR #40): its own dedicated instance, never co-located with the application, in dev and production alike, because a monitor sharing the host it watches cannot report that host dying (NFR-18). Dev's instance (2 vCPU, 2 GiB, limits scaled to ~1.78 GB) is not provisioned yet and the cross-host metrics wiring is not built.
• 7 Oct Ingestion's deployment wiring (mib-infra#52, PR #57): OSS and Qwen settings, optional, with the LLM flags off; pipeline-cycle.sh runs cleanup then correct, reports exit 75 as retry later and 78 as misconfigured. Peak memory measured: 394 MiB on PMK 60/2026 (#53, PR #59). JDIH egress recorded (production/egress-rules.tsv, scripts/check-egress.sh, PR #58), not yet in the live security group.
• 10 Oct No service holds DDL (mib-infra#16, PR #67): a migrator role per schema owner owns the tables and alone runs migrations (scripts/migrate.sh); runtime roles keep data access. Existing hosts are converted once by provisioning again; smoke-db-roles.sh asserts the split. deploy.sh migrates with the new image first and never starts the ingestion one-shot (PR #66). Dev ingestion limit 768 MiB (#62).
• 12 Oct Region and models (PRD v0.32): production is most likely a Contabo VPS in Singapore; OSS and Model Studio are in ap-southeast-1. This reverses AD-4's Jakarta: personal data leaves Indonesia (a UU PDP point for the client, NFR-15), egress control moves to the host firewall, and the Alibaba runbooks need rewriting. The existing dev host stays until moved.

The largest Phase 0 item and the one everything else waited on. One account per person, one authority, verified locally by every other service. Closed on every gate clause on 23 Aug.

Build

Registration and email verification, password hashing, asymmetric token signing with a published key, local verification with no network hop per request. Opaque refresh tokens held server-side with immediate revocation, single logout, a session list a user can revoke from. Capabilities as a token claim, resolved from plan version plus institutional membership. A separate token audience for administrators. Authenticated service-to-service calls. 13 Oct: password hashing shares one concurrency limiter so a login burst cannot OOM identity (mib-identity#35, PR #39), and the entitlements read answers 404 for an unknown user (#34, PR #38).

Gate

All four clauses pass. (1) A token issued by identity verifies in mib-regulations and mib-ai without calling identity, including refusal of another key, an HS256 token using the public key as its secret, a wrong audience and a wrong issuer. (2) Revoking a session ends it at the next refresh; the access-token TTL is the worst-case lag, 15 minutes plus 30 seconds of leeway. (3) A capability change takes effect within one token lifetime, with no restart and no re-login. (4) An administrator token is refused by public endpoints and the reverse.

Found

Four defects that building it surfaced and reading would not: a RecursionError on the only signing-key path production uses (every test ran on the ephemeral dev key); a 500 on login from a client whose address is not an IP (last_seen_ip is inet); mib-ai shipping an Alembic history it could not run (alembic was never a dependency); and script.py.mako missing its upgrade placeholder in all four migrating repos, so autogenerate wrote pass while reporting success.

Property

An administrator is a row in identity.users with a role, so "not obtainable by escalating a public token" is not structural. It holds because the audience is recorded on the session at login rather than derived from the role at refresh. Writing the role column on an account already logged in as a customer cannot promote that session. Two privilege levels, neither able to alter its own role; the last active super-administrator cannot be removed by anyone; super-administrators come only from a seeding script run on the host; every administrator write records actor, before and after in the same transaction.

Not built

No Redis layer in front of sso_sessions: revocation must be immediate, so PostgreSQL stays authoritative. PPN rate and manual quota grants were split to mib-identity#29; both depend on plans and subscriptions.

Blocks only the front end. Design system first, then screens composed from it, including the states that generate the most rework when left unspecified.

Build

Design system and brand identity as the single source of truth, tokens exported to code rather than transcribed. The phase-one journeys end to end at both breakpoints. Empty, loading, error-with-reference, quota-exhausted, unentitled and no-readable-text states designed. A print-suitable logo asset for the download watermark.

Gate

Client sign-off recorded per screen, with approved screens versioned so a later change is visibly a change. After sign-off, changes are change requests against Phase 2, not free revisions.

• 9 Oct Every approved screen sliced as a client mock on mock data (mib-web#27, closes #1 and #5, tagged v0.2.0); colour, type and page width from tokens; tier states per the client's entitlement matrix, previewed with a runtime DEMO_MODE switcher; registration and login working against identity.
• 10 Oct Cleaned for the client's review (mib-web#31, v0.2.2): no developer text on screen, JPY per 100 yen on both calculators, one shared loader on every user-started API call.
• 13 Oct Figma complete. Client feedback is minor after earlier rounds and does not block the build; each item goes into mib-web#37, and one that changes an API goes through its contract PR.

Fill the corpus. Re-specified 30 Sep 2026 (PRD v0.12, extended in v0.13): regulations come from JDIH Kemenkeu as full PDFs from 2015 (the older 54% is HTML fulltext plus companion files), and tax-court rulings from SETPP, the listed risalah only (about 34,000, 2011 to mid-2019). The previous source is retired. Text extracted from the file is the only input to search and chunking, with one decided exception: LLM-corrected text for scans whose built-in text layer is faulty (mib-ingestion#21).

Build

One adapter per source, so a source change cannot break the others. The source file stored unmodified (the per-user watermark is applied at download), raw text and display HTML to OSS under deterministic keys, the text addressed by its own content hash. Text extraction that survives page headers, footers, page numbers and hyphenation so a citation such as Pasal 1 angka 5 stays intact. Metadata and typed relations from JDIH's detail pages. A PDF with no text layer is flagged needs_ocr and alerted; OCR itself is out of scope. Optional LLM HTML cleanup as its own resumable one-shot, and LLM correction of faulty scan text behind a typo-only gate. Kurs KMK and Tarif Sanksi Bunga KMK as plain lookups. A weekly sweep, with change detection a non-blocker.

Gate

The JDIH corpus (about 8,700 regulations) and the SETPP risalah (about 34,000) ingested and searchable. Re-running creates no duplicates and no re-embeds, and text extracted twice from the same file is identical. Extracted text chunks with zero crashes on a JDIH and a SETPP sample. A revoked regulation's status changes without its own listing moving. A document whose structure is not recognised is counted and alerted, never processed silently. Metadata and relationship accuracy sampled manually and recorded.

Timeline

• 5 Oct #18 (PR #26): full_text dropped; GIN trigram indexes (pg_trgm) on title and number; no search_vector. The owner role must install pg_trgm before the next deploy. #1 (PR #27): the JDIH adapter discovers through search, feed and JDIHN, reads detail pages, downloads PDFs and pre-2015 companions, resumes after a restart. #2 (PR #28): English schema and metadata mapping. #3 (PR #29): text extraction, artifacts, needs_ocr per page with one grouped alert per run.
• 6 Oct #4 (PR #30): upsert by source_ref, which for JDIH is produk_hukum_id (OQ-38; the slug is its own column); natural key (type, number, year, issuing unit); display HTML sanitised by an nh3 allow-list. #6 (PR #31) change detection; #5 (PR #32) the sweep refuses with exit 75 while embedding drains, scheduled Saturday 00:00 WIB. PRs #33 to #38: the changed-provisions parser (#25; a partial list is unreadable, never parsed), a live robots.txt guard (#10), Kurs/Tarif rows (#22), the HTML cleanup one-shot (#19, #20) and the text-correction one-shot (#21); both exit 0, 75 or 78.
• 7 Oct #39 (PR #43): run_once runs the whole cycle, then re-reads the detail page of any stored document a new one revokes or amends (R-5). #16 (PR #44): the SETPP adapter, one request per second into putusan; an id that always fails (37402) is skipped by setting, #60 (PR #61). #7 (PR #45): a parse-volume alert against the median of the previous 8 runs. #49 (PR #53): regulation_relations is written at last, with reciprocal rows and owner tracking. #48 (PR #52): issuing unit read from current KMK numbers. #54 (PR #57): Kurs change_amount from fiskal. #55 (PR #56) and #46 (PR #62): an image-stamped promulgation date, and compound change-and-delete items.
• 8–9 Oct #66 (PR #67, provisions/5): annex, Penjelasan, structure and diganti leads; PMK 71/2026 and 63/2026 parse. #70 (PR #71, provisions/6): each run re-parses stored documents left by an older parser. Lampiran kept in the text (#68, PRs #69 and #72, jdih-extract/6, PRD v0.29): blank forms and pages past 200 per segment are left out, each with a note; PMK 7/2026 (2,003 pages) builds in about 200 MiB. mib-rag embeds annex pieces tagged section = lampiran (decided 9 Oct, mib-rag#1).
• 9–10 Oct Two dev OOMs. #73 (PR #74, v0.3.1): an annex header that is an image; memory returned between documents; a source over 100 MB is stored unextracted. #75 (PR #76, v0.3.2, mib-shared v0.10.0): a 451 MB download held in memory twice over; downloads and uploads now go through disk and that file peaks at 220 MiB.
• 10 Oct #77 (PR #78, v0.4.0): parser provisions/7 reads leads provisions/6 left unreadable (2,456 of 8,505, 65% from HTML layout tables); jdih-extract/8 reads layout tables as text; a CONVERTER_VERSION bump now re-extracts stored documents from the source kept in OSS (reextract_batch, 200 a run, [TBD]). #80, #81 (v0.4.1, mib-shared v0.11.0): a re-extracted putusan keeps one text row, and superseded OSS objects are deleted once the replacing write commits; objects left earlier need python -m mib_ingestion.jobs.prune once.
• 11 Oct First dev full run (v0.4.0 then v0.4.1, about 8 h): 8,402 regulations and 33,997 putusan re-extracted; unreadable provision lists down from 2,456 to 1,277; 0 re-extraction or OSS-delete errors; 322 item failures. #83 fixed three causes (TAP MPR out of scope, shared slug 0 rebuilt from OSS, lone surrogates); mib-infra#69 skips 115 SETPP ids that always answer 500; #70 sets both catch-up batches to 50000. Left: 148 JDIH files missing upstream, 31 natural-key collisions to review, 5 records without title or number. TC-10 (memory) still unmeasured.
• 12 Oct #84 (PR #85, jdih-extract/9): no table row starts with |; a heading cell gets its own line, a list marker joins its text, data tables keep their columns. Every HTML document gets a new content_hash on the rebuild (~8 h on dev), so the first full embedding run (mib-rag#6) goes after it. #86 (PR #87, provisions/8): a coarse article or annex reading that returns a complete set of units or refuses, marked in provision_parse_reason; the next dev run re-parses every stored list (TC-33).

The first thing a user sees. Two items carry disproportionate risk: the visitor preview, planned as the primary organic discovery channel (whether search engines may list it at all has been with the client since 2 Oct, OQ-35), and watermarked PDF delivery, which puts file-sized work on the request path.

Build

Authentication and profile screens, keyword search with filters, the list view, and the HTML detail page with preserved structure and a view watermark. The connections graph, with unresolved references shown as text rather than hidden. Visitor tiering, server-rendered and (unless OQ-35 decides otherwise) indexable, with structured metadata. Watermarked PDF streaming under a concurrency cap, with a download audit record and a per-account rate limit.

Gate

Search and detail latency targets met at the 95th percentile. The withheld portion of a visitor preview is genuinely not delivered to the client, not merely hidden. Every download is recorded. Stamping throughput and peak memory per document measured under load, and the threshold at which the design reverts written down before launch.

• 27 Aug Started ahead of ingestion, querying whatever rows exist: list, detail and keyword search (#1, PR #19); semantic search gated by regulations.search.semantic (#2, PR #20, with mib-retrieval#10); the HTML view with schema.org/Legislation data, a visitor text truncated by an HTML-aware truncator to preview_char_limit characters before any markup is built (#3, PR #21). That PR forced the search and internal text endpoints off full_text.
• 2 Oct (PRD v0.14) The semantic-search mode on the list is switched off (no client agreement; no plan grants the capability); search by meaning is part of the AI chat. Every fallback PDF is rendered before go-live (#5).
• 5 Oct Keyword search narrowed to title and number (trigram indexes); #27 re-scoped.
• 8 Oct Client's feature list: tax treaties are public, watermarked, searched by country and title, not in the AI chat (#31, mib-web#19); Kurs and Tarif get public pages with Excel export (#32, mib-web#21); Premium gates Compile & Compare (#33, mib-web#22), which reverses AD-8.
• 13 Oct #28 (PR #29): the read model is the English schema, search matches title, number and number_display (a % or _ is literal), so live search works; ranking and the exact-number lookup stay in #27 and relevance still sorts newest first. The same PR serves the corrected text with its own hash, the Kurs/Tarif lookup and exclude_from_rag. #24 (PR #47) and #25 (PR #48) fixed.
• 13 Oct Contract-first. The public API is OpenAPI written by mib-web from its pages, in docs/contracts, and the backend adapts. C1 is frozen at 1.0.0 (PR #46) and corrected in 1.0.1 (#51). It records partial revocation, a bare number beside number_display, optional access on downloads (the watermark and the audit row need the caller), and only the putusan fields the table stores (case number, abstract, year, tax-type code, risalah date; no title, outcome or stage; no peninjauan-kembali list).

The whole paywall rests here: with the corpus free to any registered account, AI quality and unit cost are the commercial proposition. And the AI part has its own contractual acceptance test.

Build

Chunking that respects document structure before falling back to length. The embedding state machine, driven by one staleness predicate covering new, changed and pipeline-migrated regulations, with rows claimed so concurrent workers cannot collide. The backfill, costed and timed before it runs; it advances only during maintenance windows, so its finish depends on how many windows the client accepts as much as on throughput. Retrieval, two-stage ranking and caching. The chat: one sorting call then fixed steps, exact lookup for a named document, facts (status, dates, relations, changed provisions) filled in by code from the database, every number checked before the AI's text is shown, the whole answer at once with no streaming. Summaries stored once per document version and shared. Quota per answered message: 2 a day for visitors and Gratis, 50 for Premium (the client's feature list, 8 Oct; replaces the rolling window and the lifetime grant, mib-ai#2).

Gate

The vector index type chosen by benchmark, recording resident index size and not only recall and latency. Quota correct at the boundaries, including the daily reset (00:00 WIB) and a guest's count carrying to a new account for the rest of the day. A Qwen outage degrades to lexical search and consumes no quota. Our own run of the UAC passes before the client's UAT (mib-ai#20).

• Built Started ahead of Stage 1 on purpose: chunking needs text, not a database, and was measured on 54 scraped documents; the 296-document sample then found a crash (mib-rag#23). Structure-aware chunking and the Qwen embedding client; the claiming machinery; the worker loop (mib-rag#27); retrieval (mib-retrieval#1); POST /api/v1/ai/summary (mib-ai#1); superseded-regulation gating (mib-ai#9). None of it has run against a real corpus.
• 24–26 Aug diubah refused new summaries and dicabut warned instead; revised 26 Aug so a revoked regulation generates with a warning (the text itself is not in dispute). Reversed 30 Sep: an amended regulation also gets a prominent warning, not a refusal (mib-ai#17, not built, so the code still refuses). A regulation's relations are whole-document only (FR-IN-03), so the refusal was regulation-wide, not scoped to a section.
• 2 Oct Summaries stored once per document version and shared (mib-ai#3); putusan summaries in scope (mib-ai#18); TC-RAG-01's changed-provisions list is read by code at ingestion (mib-ingestion#25), never written by the AI; putusan vectors at half precision (mib-rag#35); a retrieval request names exactly one corpus (FR-RS-18).
• 13 Oct mib-rag#36 (PR #37): exclude_from_rag is carried like needs_ocr; the sync drops an excluded regulation's chunks; served_content_hash makes a correction landing after the first embedding re-embed that document once.

Plans as configurable, versioned data rather than code, so pricing and packaging change without a release. Invoicing carries tax obligations from day one.

Build

The plan record with its capability set, versioned rather than edited in place. The payment integration behind a provider-agnostic adapter, with signature verification and idempotent webhook handling. Entitlement resolution from plan version and membership. Invoices bearing the configured tax rate, with buyer details snapshotted at issue and a gapless number sequence. Institutional seats on one subscription. Checkout, and cancel with a reminder and a questionnaire.

Gate

A price change leaves existing subscriptions on what they paid, and applies at renewal with notice. An issued invoice is never mutated; a correction is a new document. Replaying a webhook is a no-op. Seat exhaustion refuses the next member with a defined error rather than silently granting access.

Partly needed in Phase 1, since reviewing ingestion output cannot wait for the full build. The authentication under it already exists (audience, both roles, seeding script, audit log, from Stage 0.3), so what is left is the surface and the editors rather than the security model. The surface is part of mib-web on admin.DOMAIN rather than its own application (decided 23 Aug): one fewer container on a 2 GiB host, at the cost of a credential that can suspend accounts living in the process that serves public pages (.github#7).

Build

Regulation CRUD with a manual-lock marker so corrections survive the next crawl. Ingestion history and retry. User and subscription management. Content editing bounded to text, images and banners, not a page builder. Plan, capability and tax-rate editing behind the super-administrator role, every change audited before and after. Poller and sweep configuration.

Gate

A day-to-day administrator cannot alter prices, capabilities or their own role. This is already true and tested as of Stage 0.3; what this gate adds is that it stays true once there are prices and capabilities to alter. Capability names come from a fixed list, never free text, and the impact of a change is shown before it is saved.

Deliberately unspecified. The requirements captured so far are an outline from the architecture diagram, not an agreed specification, and must be re-specified before anything here is estimated or scheduled (mib-lms#1). It benefits from identity already being built: the LMS verifies existing tokens rather than building authentication. mib-lms is a scaffold with no routes.