Operations

Day-2 operations for the QSign stack: backups, monitoring, logging, upgrades, scaling, and maintenance. Commands assume you run from the compose directory (/opt/qsign).

Health & monitoring

• Liveness endpoint: GET https://sign.example.com/health → 200. Point your uptime monitor at it.
• Container health: docker compose ps (the DB has a healthcheck; add your own for others if desired). Set restart: always (as in the sample) so containers self-recover.
• Quick smoke test after any change:

  curl -sf https://sign.example.com/health
  curl -sf https://sign.example.com/backend/othercompanyapi/openapi.json >/dev/null && echo API-OK

• Metrics to watch: document-storage volume usage, MySQL size, CPU during conversion/OCR, Redis availability (the API throttle uses it — it fails open, but a Redis outage disables rate-limit enforcement), and email-send failures in the logs.

Logging

• Application logs go to the backend’s /var/log (mounted to the qsign_logs volume) and to container stdout. View with:

  docker compose logs -f qsign-backend
  docker compose logs --since 1h nginx esign

• Forward container logs to your central logging (journald/syslog driver, or a log shipper) per your standards. Configure rotation on the qsign_logs volume.
• The backend records an activity/audit trail in the database (see 08-security-compliance) — independent of these logs.

Backups

Three things must be backed up; restore requires the first two (Solr is rebuildable).

1. Database (MySQL) — required

# Backup (logical dump):
docker compose exec db sh -c \
  'exec mysqldump -uroot -p"$MYSQL_ROOT_PASSWORD" --single-transaction --routines qsign' \
  | gzip > backups/qsign-db-$(date +%F_%H%M).sql.gz

# Restore:
gunzip -c backups/qsign-db-YYYY-MM-DD_HHMM.sql.gz | \
  docker compose exec -T db sh -c 'exec mysql -uroot -p"$MYSQL_ROOT_PASSWORD" qsign'

Schedule the backup daily (cron/systemd timer) and ship the dumps off-host.

2. Document storage — required

The qsign_storage volume holds all original/converted/signed/stamped documents — this is the largest and most critical data. Back it up with your volume/snapshot tooling, e.g.:

docker run --rm -v qsign_qsign_storage:/data -v "$PWD/backups":/backup alpine \
  tar czf /backup/qsign-storage-$(date +%F).tar.gz -C /data .

Prefer filesystem/array snapshots or an off-host sync (e.g. restic/rclone to object storage) for large volumes. Keep DB and storage backups time-aligned so a restore is consistent.

3. Solr index — optional (rebuildable)

Solr can be rebuilt from MySQL + storage:

docker compose exec qsign-backend /opt/venv/bin/python manage.py reindex_solr

Back it up only if rebuild time is a concern.

Restore drill

Periodically validate restores on a staging host: restore the DB dump + storage tarball into a fresh stack, start it, reindex Solr, and confirm a previously-signed document opens and verifies. An untested backup is not a backup.

Upgrades

QSign ships as new container image versions. Standard procedure:

# 1. Back up DB + storage FIRST (see above).
# 2. Note current image tags (for rollback):
docker compose images

# 3. Pull / load the new images Quoqo provides, then update the tags in docker-compose.yml
#    (or your .env image-tag variables).

# 4. Recreate the app containers (DB/Redis/Solr unchanged):
docker compose up -d qsign-backend qsign-frontend esign qsign-esign-api nginx
#    The backend runs DB migrations automatically on start.

# 5. Verify:
docker compose logs -f qsign-backend     # watch migrations + boot
curl -sf https://sign.example.com/health

Rollback: if the health check fails, revert the image tags to the previous version and docker compose up -d again. If a migration applied and you must roll back the schema, restore the pre-upgrade DB dump. Always read Quoqo’s release notes for migration or config changes before upgrading; test upgrades on staging first.

TLS certificate renewal

• Let’s Encrypt (bundled certbot): auto-renews; ensure port 80 stays reachable for ACME and reload nginx after renewal (the sample certbot loop handles renewal; reload nginx on a schedule or via a renew-hook).
• Own/internal CA: replace the cert files before expiry and docker compose restart nginx. Track expiry in your monitoring.

Antivirus definitions

The clamav container updates its virus definitions automatically (freshclam). Confirm it reports healthy and has internet egress (or mirror definitions internally for air-gapped installs).

Scheduled jobs (cron)

In-container cron (registered via manage.py crontab add) handles daily reminder emails, Aadhaar-quota refunds, and subscription housekeeping. After an image upgrade, re-run crontab add if the schedule changed. Verify with docker compose exec qsign-backend /opt/venv/bin/python manage.py crontab show.

Scaling & high availability

• Vertical: increase host CPU/RAM (conversion/OCR/signing are the heavy paths).
• Horizontal: the backend and frontend are stateless and can run multiple replicas behind a load balancer, sharing the same MySQL/Redis/Solr/storage. Redis-backed throttling stays correct across replicas.
• Data tier HA: run MySQL as a managed/replicated instance and Redis/Solr clustered if your availability targets require it. Use shared/network storage (or object storage) for the document volume when running multiple backend hosts.

Routine maintenance checklist

• Daily: DB dump + storage backup shipped off-host; backup job alerts on failure.
• Daily/weekly: review error logs (and Sentry, if enabled).
• Weekly: storage + DB growth trend; capacity headroom.
• Monthly: test-restore drill on staging; rotate secrets per policy.
• Before each upgrade: backup, read release notes, test on staging, plan rollback.
• Watch certificate expiry and ClamAV health.