Node dataset directories
DACP nodes publish local catalogs; Union reads approved members online and exposes its own DACP namespace using short names. SDKs use the dacp.catalog Action to obtain a Directory with completeness fields. ListFlights fails for incomplete catalogs. See Union proxy service.
The Union website has three independent pages:
/: operations dashboard with jobs, execution states, DAGs, policies and attempt records./nodes: graphical member nodes with health, capacity, startup time, uptime and node website links. Status updates every five seconds. Expanding a node reads its local dataset names and URIs inline with a five-second timeout. Cards expand independently and support manual refresh and retry; dataset lists are not polled./datasets: the data catalog merges approved member catalogs online and displays Union-assigned publicnamevalues such asds-k7m2-9wqr, JSON descriptions and Union URIs. This is the same namespace returned to clients bydacp.catalog. Filtering and manual refresh are available. Unreachable members produce an explicit incomplete-directory warning rather than presenting partial results as complete.
Local names and Union public names belong to different namespaces. Datasets have no separate id attribute. Union persists name mappings for stable public names, not dataset lists. Loaded local directories stay only in the current browser page; refreshing the node list clears them.
flowchart LR
Client --> Node["Node public catalog"] --> Local["Local Dataset names"]
Client --> Union["Union public catalog"]
Union --> A["Read member A catalog online"]
Union --> B["Read member B catalog online"]
Union --> Public["Stable public names and original descriptions"]
A -.-> Partial["Report incompleteness when unavailable"]
B -.-> Partial
Access boundaries
Directory, Item-list and Item-metadata requests allow anonymous TLS access. Data streams and queries require a reader or runner subject role. See Union proxy service for management Web access boundaries. Union uses its own certificate to query the selected configured member and verifies the member identity. Arbitrary remote addresses are not accepted. Directories show names, URIs and JSON descriptions without reading Items or data contents. Published data requires a reader or runner role; execution requires runner.
Administrators: static descriptions
datasets:
orders:
provider: mixed
description: |-
{"title":"Orders","description":"Order data","owner":{"team":"sales"},"fields":["id","amount"],"updated":"daily"}
items:
/records:
file: /srv/data/orders.csv
description is a string containing one valid JSON object. It requires a nonempty string title and a string description (which may be empty); other fields and nested structures are user-defined. It is separate from provider source and items settings. When omitted from static configuration, it defaults to {"title":"dataset-name","description":""}. Duplicate JSON keys are rejected. The string is limited to 32768 characters. Plain text, arrays, scalars, malformed JSON, multiple JSON values and YAML mappings are rejected. Public descriptions must not contain credentials or private storage settings. A complete node configuration still requires TLS and other fields.
Dynamic directories
Reading a directory does not open data streams. For catalog configuration, the description JSON contract and provider examples, see the XStream catalog guide. Union preserves the original description of a renamed public Dataset.
Shell
The shell connects to DACP nodes or a Union providing DACP services. The startup argument must be a server address, not a Dataset URI:
dacp-shell dacp://localhost:18861
Supply certificates through DACP_CA, DACP_CERT, DACP_KEY, or --ca, --cert, --key.
dacp-shell> datasets
dacp-shell> use orders
dacp-shell> ls /
datasets accepts no arguments and shows row numbers, dataset names, and descriptions in the columns #, Name, and Description (JSON), without URIs. On startup or connect dacp://host:port, the shell automatically selects and prints the sole Dataset when the complete catalog contains exactly one Dataset. With zero or multiple Datasets, select one with use NAME or use NUMBER before ls, count, cat, SELECT, or run. use selects a Dataset on the current server; metadata is anonymous and data reading requires reader or runner. Union uses automatically assigned public short names, selected directly with use NAME. Incomplete catalogs never trigger automatic selection. Row numbers start at 1 and refer to the latest datasets output, for example use 1. Running datasets updates the mapping; reconnecting clears it. Numeric arguments are interpreted as row numbers, with out-of-range values rejected.
Python
import json
from pathlib import Path
from dacp import Source
credentials = dict(
tls_root_certs=Path("examples/certs/ca.crt").read_bytes(),
cert_chain=Path("examples/certs/client.crt").read_bytes(),
private_key=Path("examples/certs/client.key").read_bytes(),
)
# Connect directly to dacp-server.
for dataset in Source.datasets("dacp://localhost:18861", **credentials):
print(dataset.name, dataset.description, dataset.uri)
print(json.loads(dataset.description).get("owner", {}).get("team"))
The Scala client uses Source.datasets(origin, tls) to query nodes directly. Node FlightInfo.app_metadata contains the DataSet protobuf; descriptors use datasets/<name>.
Validation
(cd dacp-server && sbt 'testOnly dacp.DataCatalogSpec dacp.DacpServerSpec')
(cd dacp-union-server && sbt test)
(cd dacp-client-scala && sbt test)
PYTHONPATH=dacp-client .venv/bin/python -m pytest dacp-client/tests
Union rewrites only the public name and URI. It retains the member’s original description JSON string exactly, including title, description, custom fields and whitespace, without generating a new title or reserializing the object.