DataCatalog Integration, Source Connections, and Payload Conventions

DataCatalog integration is not only a UI convenience in FlowMaker; it is the configuration and semantic backbone for robust multi-worker flows.

Why DataCatalog should be central

Use DataCatalog as the single source of truth for:

Connection credentials and endpoint details.
Database/dataset/bucket/topic coordinates.
Extra protocol-specific options that should not be hardcoded in each worker.
Tag/entry identity and metadata used downstream.

This reduces config drift between source, mapper, and sinks, and avoids repeating sensitive connection details across multiple worker forms.

[NOTE!lightbulb/CENTRALIZED CONFIG STRATEGY] Keep credentials and infrastructure coordinates in DataCatalog source connections whenever possible. Worker options should mostly reference IDs (connection IDs, entry IDs), not duplicate secrets or endpoint strings.

Source connection + entry selection model

Common frontend pattern:

Select a DataCatalog source connection (DCSourceConnection).
Select one or more entries/tags (DCCatalogEntry / DCCatalogEntryPicker) scoped to that source connection.
Persist IDs into worker options.

This is visible in real worker frontends such as OPC-UA config forms where connection details are resolved from DataCatalog and tag selection is entry-based.

[INFO!hub/OPC-UA EXAMPLE] In OPC-UA frontend config, the form explicitly states endpoint/security/credentials are stored in the source connection, while selected tags come from DataCatalog entries.

Canonical payload convention for telemetry in flows

For interoperability across pipes/sinks, keep telemetry properties flat at the payload root and reserve special keys:

Root properties for measured tags (for example temperature, pressure, flow_rate).
$$meta for per-property metadata/mapping.
$$timestamp for event time.

Canonical shape:

{
  "temperature": 21.7,
  "pressure": 1.8,
  "$$meta": {
    "temperature": { "entryId": "dc-entry-temp" },
    "pressure": { "entryId": "dc-entry-pressure" }
  },
  "$$timestamp": "2026-04-10T16:00:00.000Z"
}

[WARNING!schedule/ROOT-PROPERTY CONTRACT] If tags are deeply nested instead of flat at root, automatic mapper/sink behaviors that resolve properties by key name can fail or silently skip data.

How mapper and sinks use this contract

DataCatalog Mapper

datacatalog-mapper appends/overwrites $$meta by mapping configured property names to entry IDs.

[NOTE!account_tree/MAPPER BEHAVIOR] The mapper only emits metadata for properties that are both configured and present in the incoming payload.

MQTT source/sink and metadata-aware publish

MQTT source emits root property values and includes both $$meta and $$timestamp.
MQTT sink can enrich from $$meta and auto-route publications using catalog entry metadata.

Timeseries sink

Timeseries sink enriches from $$meta, groups records by catalog entry connection params, and resolves event time from configured mapping with fallback to $.$$timestamp.

[NOTE!lightbulb/TIMESTAMP FALLBACK] If sink-specific time mapping is absent/invalid, $.$$timestamp acts as the canonical fallback path.

Practical guidance

Keep tag names stable and flat at payload root.
Treat $$meta as machine-consumable mapping state, not UI display text.
Emit a consistent $$timestamp at source or earliest normalization step.
Use DataCatalog entry IDs as durable references; avoid free-form downstream mapping strings when IDs are available.

[WARNING!shield/DRIFT RISK] Duplicating connection details in each worker option increases security and maintenance risk; one credential rotation can break multiple boxes inconsistently.

References

/home/dja/dsd/dev/boxes-repo/flowmaker-workers/workers/opc-ua-client/frontend/src/source/ConfigForm.svelte
/home/dja/dsd/dev/boxes-repo/flowmaker-workers/workers/datacatalog-mapper/src/datacatalog-mapper.ts
/home/dja/dsd/dev/boxes-repo/flowmaker-workers/workers/mqtt-client/src/mqtt-source.ts
/home/dja/dsd/dev/boxes-repo/flowmaker-workers/workers/mqtt-client/src/mqtt-sink.ts
/home/dja/dsd/dev/boxes-repo/flowmaker-workers/workers/timeseries-workers/src/timeseries-data-sink.ts