Getting Started
Architecture
NNIPA is built as a set of purpose-built layers around a single shared lakehouse, not one monolithic database or a patchwork of disconnected tools. This page walks through how those layers fit together, from the moment data lands to the moment an analysis comes back.
System layers
Every feature in NNIPA is built on top of the same six layers — none of them are optional add-ons bolted on afterward.
Connect
Files, databases, CDC streams, and REST/GraphQL APIs all land through the same connection layer, so everything downstream treats them identically regardless of where the data came from.
Lakehouse storage
Structured data is stored in an Apache Iceberg lakehouse — every change keeps a full snapshot history, so nothing is silently overwritten.
Query engine
A Trino-based query engine is the single execution surface for every read — dashboards, analyses, and exports all run through it, which is also where tenant scoping is enforced consistently.
Catalog & quality
New tables are discovered and documented automatically, then checked against quality rules — so a dataset is trustworthy and searchable before anyone has to configure it by hand.
Intelligence layer
A semantic layer builds an understanding of each dataset's columns, types, and likely relationships, and feeds that into analysis suggestions, the AI Analyst, and Ask Questions.
Digitization
Unstructured documents follow a parallel path — OCR extraction, confidence scoring, and human review — before joining the same lakehouse as everything else.
How a request flows
Take running a statistical analysis as an example — every other computed result (an ML prediction, a saved query, a workflow step) follows the same shape.
- 1
You choose a dataset and an analysis
Whether it's a statistical method, an ML model, or a saved query, the request starts from the Catalog or Analyze section.
- 2
The request resolves who's asking
Every request carries your identity and your tenant. Nothing downstream re-derives that from scratch — it flows through the whole chain.
- 3
Computation reads through the query engine
The statistical engine and ML pipeline never open a direct connection to storage — they read through the same query engine everything else uses, so tenant scoping applies no matter which feature is asking.
- 4
Work runs as a background job
Analyses and training runs execute in dedicated worker processes. Large datasets don't block your browser — you get a job you can check back on.
- 5
Results are stored, not recomputed
The result, and the exact query or configuration that produced it, are saved. Reopening a finished analysis is instant — it never runs again just because you looked at it.
Multi-tenancy & isolation
Every dataset, analysis, workspace, and API key belongs to exactly one tenant, and that scoping is enforced at the same layer that runs every query — not just checked at the edge and trusted afterward. A user with access to two organizations sees two entirely separate catalogs, job queues, and result histories; nothing about one is reachable from the other, even by direct link.
Within a tenant, five built-in roles — owner, admin, analyst, viewer, and member — control what a person can see and do, and administrators can layer custom permissions on top of those for anything the built-in roles don't cover. See Tenant Isolation and Authorization for the full model.
- Background computation — statistical jobs, ML training, workflow runs — happens in dedicated worker processes, separate from the servers handling your requests, so a long-running analysis never slows down anyone else's page.
- The catalog re-syncs on its own, so schema changes in a connected source show up without anyone re-registering the table by hand.
- Structured data and digitized documents both end up in the same lakehouse, so a table built from a CSV upload and one extracted from scanned documents can be joined and analyzed the same way.