Appearance

Consolidated Logging in Compose ​

DeltaFi's Docker Compose orchestration includes a consolidated logging system that collects, processes, and stores logs from all DeltaFi services. This system provides easy access to logs through a web interface and ensures proper log rotation and management.

Overview ​

The logging stack consists of these components:

  1. Vector - Collects and processes logs from all DeltaFi containers
  2. VictoriaLogs - Stores logs for searching in the DeltaFi UI and in Grafana
  3. Logrotate - Manages rotation and retention of the text log files

VictoriaLogs and the logrotate text files are independent log outputs. Vector ships logs to whichever outputs are enabled, so you can run either, both, or neither.

Enabling Logging ​

Logging is enabled by default in Compose deployments, with Vector, VictoriaLogs, and the logrotate text files turned on. deltafi.logs.enabled is the master switch; when it is false, no log collection runs and the Vector, Logrotate, and VictoriaLogs containers are not started:

yaml
deltafi:
  logs:
    enabled: false

Vector and each log output also have their own toggles in site/values.yaml:

yaml
enable:
  vector: true             # Vector log shipping (same flag as Kubernetes)
  victorialogs: true       # VictoriaLogs store and the UI "Logs" viewer (same flag as Kubernetes)

deltafi:
  logs:
    enabled: true          # Master switch
    logrotate:
      enabled: true        # Text log files in the logs directory

Vector only runs when at least one output is enabled. Setting enable.vector: false stops log shipping, which also stops the logrotate text files; VictoriaLogs keeps running so previously collected logs remain searchable, as on Kubernetes. See Site Configuration - Observability Components for the other observability toggles.

Log Collection ​

Vector Log Aggregation ​

Vector collects logs from all Docker containers that have the deltafi-group label. This ensures that only DeltaFi-related containers have their logs collected, preventing logs from unrelated containers on the system from being included.

Vector processes logs in the following way:

  1. Collection: Collects logs from all containers with the deltafi-group label
  2. Parsing: Parses JSON-formatted logs and extracts metadata
  3. Routing: Separates audit logs from regular application logs
  4. Output: Writes logs to the local filesystem and/or ships them to VictoriaLogs

VictoriaLogs ​

When enable.victorialogs is true, a deltafi-victorialogs container stores logs under {dataDirectory}/victorialogs. Vector ships each log entry with the same fields used on Kubernetes, so the same queries and dashboards work in both modes:

  • namespace - Always deltafi
  • app - The container's app label (plugins and core actions), otherwise the container name
  • pod / container - The container name
  • stream - Output stream (stdout/stderr)
  • type - ACTION for action logs, AUDIT for audit logs
  • Fields from JSON log bodies (level, loggerName, action, user, etc.)

Retention ​

VictoriaLogs deletes the oldest logs when either limit is reached. The keys are the same as the Kubernetes chart's:

yaml
victorialogs:
  server:
    retentionPeriod: 13d                 # time-based retention (default: 13d)
    retentionMaxDiskUsagePercent: 50     # cap at a % of the filesystem holding the DeltaFi data directory (default: 50)

The disk percentage applies to the whole filesystem that holds the DeltaFi data directory, which is shared with Postgres and content storage. Keep it low enough to leave room for DeltaFile content.

Log Output Format ​

Regular Logs ​

Regular application logs are written to individual files per container:

/logs/<container-name>.log

Each log entry is a flattened JSON object containing:

  • level - Log level (INFO, WARN, ERROR, etc.)
  • message - The log message
  • timestamp - Timestamp of the log entry
  • container - Container name
  • stream - Output stream (stdout/stderr)
  • Additional fields from the original log (loggerName, threadName, etc.)

Example log entry:

json
{
  "container": "deltafi-core-scheduler",
  "level": "WARN",
  "loggerName": "org.deltafi.core.delete.DiskSpaceDelete",
  "message": "No DeltaFiles deleted -- disk is above threshold",
  "stream": "stdout",
  "threadName": "threadPoolTaskScheduler-6",
  "timestamp": "2025-12-03T00:05:32.095521511Z"
}

Audit Logs ​

Audit logs are consolidated into a single file:

/logs/audit.log

Audit logs are automatically detected (logs with loggerName: "AUDIT") and written separately. Each audit log entry contains:

  • level - Log level
  • message - The audit message
  • timestamp - Timestamp of the audit event
  • user - User who triggered the action
  • Additional audit-specific fields

Example audit log entry:

json
{
  "level": "INFO",
  "message": "created event f2864a64-50cf-45fd-ac89-ce52476fcf9b",
  "timestamp": "2025-12-02T23:48:43.694Z",
  "user": "TUI"
}

Viewing Logs ​

VictoriaLogs Web Interface ​

When VictoriaLogs is enabled, the Logs item in the DeltaFi UI Administration sidebar opens the VictoriaLogs web UI (/select/vmui/), the same as on Kubernetes. It is visible only to administrators. Use it to search and filter logs with LogsQL, for example:

app:deltafi-core-scheduler level:ERROR

Grafana Log Dashboards ​

When both VictoriaLogs and dashboards (deltafi.dashboards.enabled) are enabled, Grafana is provisioned with a VictoriaLogs datasource and the Logs, Log Overview, Audit Logs, and Action Logs dashboards.

Direct File Access ​

When the logrotate output is enabled, logs are also available directly on the filesystem at the location specified by LOGS_DIR (typically ~/deltafi/data/logs). You can access them using standard command-line tools:

bash
# View logs for a specific container
tail -f ~/deltafi/data/logs/deltafi-core-scheduler.log

# View audit logs
tail -f ~/deltafi/data/logs/audit.log

# Search logs
grep "ERROR" ~/deltafi/data/logs/*.log

Log Rotation ​

Logrotate automatically manages log file rotation and retention to prevent disk space issues.

Configuration ​

Log rotation can be configured in site/values.yaml:

yaml
deltafi:
  logs:
    enabled: true
    logrotate:
      interval: daily          # Rotation interval: hourly, daily, weekly, monthly
      schedule:
        max_size: 50M          # Rotate when file reaches this size
        keep: 30               # Keep regular logs for 30 days
      audit_schedule:
        max_size: 100M         # Rotate audit logs when file reaches this size
        keep: 365              # Keep audit logs for 365 days

Default Behavior ​

  • Regular logs: Rotated daily or when they reach 50M in size, kept for 30 days by default
  • Audit logs: Rotated daily or when they reach 100M in size, kept for 365 days by default
  • Rotation occurs based on whichever condition is met first (time interval or size threshold)
  • Old log files are automatically compressed and removed after the retention period

Log Storage Location ​

Logs are stored in the directory specified by the LOGS_DIR environment variable. The default location is {dataDirectory}/logs, where dataDirectory is set in your TUI config.yaml file and defaults to {installDirectory}/data (run deltafi config to see the resolved value, or deltafi config --json to read it programmatically).

For example, if your config.yaml contains:

yaml
dataDirectory: /Users/myuser/deltafi/data

Then logs will be stored at /Users/myuser/deltafi/data/logs. In a standard installation with installDirectory: ~/deltafi, logs will be at ~/deltafi/data/logs.

The logs directory will be created automatically when logging is enabled.

Vector Configuration ​

Vector's configuration ships with the DeltaFi orchestration files in orchestration/compose/settings/vector/ and is split by output:

  • vector.yaml - Container log collection (only deltafi-group labeled containers) and the health check API
  • vector-files.yaml - JSON parsing, audit log routing, and the text file output (loaded when the logrotate output is enabled)
  • vector-victorialogs.yaml - Field normalization and the VictoriaLogs output (loaded when VictoriaLogs is enabled)

The TUI selects the files on each deltafi up and passes them to Vector through the VECTOR_CONFIG environment variable. These files are managed by DeltaFi and are replaced on upgrade, so local modifications are not recommended.

Health Monitoring ​

Vector exposes a health check endpoint on port 8686. The health status can be checked:

bash
# From the host
curl http://localhost:8686/health

# From within the Docker network
curl http://deltafi-vector:8686/health

Troubleshooting ​

Logs Not Appearing ​

If logs are not appearing in VictoriaLogs or the log files:

  1. Verify logging is enabled: Check deltafi.logs.enabled, enable.vector, deltafi.logs.logrotate.enabled, and enable.victorialogs in site/values.yaml
  2. Check Vector container status: docker ps | grep vector
  3. Verify container labels: Ensure containers have the deltafi-group label
  4. Check Vector logs: docker logs deltafi-vector
  5. For VictoriaLogs, check the store is running: docker ps | grep victorialogs and docker logs deltafi-victorialogs

Missing Container Logs ​

If a specific container's logs are missing:

  1. Verify the container has the deltafi-group label
  2. Check that the container is running: docker ps
  3. Verify Vector is collecting from that container: Check Vector logs for exclusion messages

Disk Space Issues ​

If VictoriaLogs is consuming too much disk space, lower victorialogs.server.retentionPeriod or victorialogs.server.retentionMaxDiskUsagePercent.

If log files are consuming too much disk space:

  1. Check logrotate configuration and retention settings
  2. Verify logrotate is running: docker ps | grep logrotate
  3. Manually rotate logs if needed: docker exec deltafi-logrotate logrotate -f /etc/logrotate.conf

Contact US

Copyright © 2021-2025 DeltaFi. All rights reserved.