Troubleshooting

This guide organizes troubleshooting by symptom. General principle: collect evidence with read-only queries before making changes. At each step, record observations, commands and exit codes, and differences between expected and actual behavior.

Device unreachable (SSH connection fails)

Investigate each layer separately:

1. Verify the address against the device inventory; similar addresses are not interchangeable.
2. Check address resolution, routing, SSH port connectivity, and authentication in sequence.
3. If a previously reachable device suddenly disconnects, correlate the event with recent updates or reboots. If necessary, ask on-site personnel to check the serial console or physical condition.

Boot failures or repeated restarts

1. Collect boot output, previous-boot logs (journalctl -b -1), watchdog reset reasons, and filesystem errors.
2. Identify the failing stage: bootloader, kernel, system switching, service startup, watchdog reset, or application-triggered action.
3. Check bootcount. Consecutive failures trigger automatic recovery: after thresholds are reached, recovery mode and partition rollback occur in sequence. See A/B systems and switching.
4. Check current recovery state with jelina-info and jelina-auto-recovery --state.

Do not manually switch partitions or roll back before identifying the cause, as this can disrupt automatic recovery.

System enters a degraded state

systemctl is-system-running
systemctl --failed --no-pager

Inspect failed services individually, then check core services. Failure of an optional platform-specific service does not necessarily indicate a system fault; one-shot services being inactive after completion is also normal.

Update appears stalled

1. Check update status and progress with jelina-info --get ota_status --json and jelina-local-ota -s.
2. Check for updates awaiting confirmation. With local confirmation enabled, waiting is an intended policy state, not a hang.
3. Inspect the device agent service and logs.
4. Check disk space and update installer status and processes.
5. Check package accessibility, system clock, DNS, and certificates.
6. Determine whether installation has completed and is simply waiting for a reboot or remount to activate.

During troubleshooting, do not resubmit or reconfirm updates repeatedly, and do not delete pending-reboot state markers.

Application or container unavailable

Check each stage separately: desired state, image download, container creation, process health, ports and firewall, and application status. Common commands:

docker ps                # Container status
docker logs <container-name>      # Container logs

Do not manually recreate system-managed containers or services before preserving evidence.

Disk full or root filesystem read-only

1. Compare storage and inode usage on root and data partitions with df -hT and df -i.
2. Check for temporary root mode with jelina-enter-tmpfs-root --status; see storage and data partition.
3. Distinguish among a deliberately read-only system partition, a data partition fault, and an overlay that failed to mount correctly.

Do not delete unfamiliar system or package directories. Do not perform business operations in temporary root mode.

Registration fails or cloud status is offline

1. Check registration with jelina-manual-register --check; see device registration and connectivity.
2. Check whether offline mode is enabled.
3. Check system clock, DNS, routing, certificates, and cloud endpoint accessibility.
4. Inspect registration and device agent service status and errors.
5. Distinguish incomplete initial activation, invalid credentials, network faults, and cloud rejection.

Service fails to start

Check startup conditions, program file existence and permissions, script interpreters, and readiness of required mounts, in that order.

Time or DNS errors

• Incorrect time often appears as certificate validation or cloud authentication failures. Compare system time, hardware clock, and time synchronization service logs.
• For DNS errors, check resolver service status and network configuration. Do not manually overwrite system-generated resolver configuration files.

GUI unavailable

An SSH terminal usually lacks a desktop session environment. Empty display variables do not indicate a GUI failure:

1. Check the interface master switch and the scope of any disabling settings.
2. Inspect interface instances and sessions with jelina-ui-ctl list and jelina-ui-ctl sessions.
3. Check active graphical login sessions and inspect interface processes and logs in the target user’s environment.

Historical evidence for restart issues

Some devices enable archived performance data and persistent logs to reconstruct the state before a restart:

journalctl --list-boots              # Boot list
journalctl -b -1 -p warning..alert   # Warning logs from the previous boot
ls -1t /var/log/atop | head          # Historical performance archives

Determining the cause of an unexpected restart requires correlating boot history, previous-boot logs, watchdog reset reasons, uptime, and update/partition-switching history.

Related documentation

• FAQ — Frequently asked questions
• A/B systems and switching — Boot count and rollback
• Watchdog — System health protection
• System status and information — Status query commands