Suggestions for log aggregation for proxmox containers
from disobey2623@lemmy.dbzer0.com to selfhosted@lemmy.world on 13 Mar 10:52
https://lemmy.dbzer0.com/post/65276809

Heya.

I’m still pretty new to the homelab scene, so the more detail you can add the better. I’d like to add some sort of log aggregation tool, something like Elastic, where I can go to look at logs from any of my systems that aren’t working, or just make sure I don’t miss any errors.

Pretty much everything I run is set up as a Proxmox LXC from Proxmox helper scripts, which most of the time means it’s running as a systemctl service. Sometimes they run in Alpine instead, and a few of my apps also run in Docker.

What’s a good app to aggregate logs from those sources? I’ve heard of Prometheus, Grafana and Loki but not sure if they do what I’m after, they seem pretty overwhelming and more focused on metrics, whereas I want to be able to search for and view logs. I’d appreciate if you also mention the basic steps to send the logs from each container to said app.

#selfhosted

threaded - newest

mangaskahn@lemmy.world on 13 Mar 11:11 next collapse

Greylog is a syslog aggregator that might do what you’re looking for.

redlemace@lemmy.world on 13 Mar 11:48 collapse

I use syslog on everything that can send it. All forward to a centralized rsyslog which stores directly into postgres. Grafana nativly can read from that. Super lightweight, super flexible. (I use 1 cpu and 2G mem and it works better than graylog which needed 4 cpu’s and 16 G ram)

slazer2au@lemmy.world on 13 Mar 12:08 collapse

Do you have pointers on how to get that up an going?

redlemace@lemmy.world on 13 Mar 14:26 collapse

this is the main pointer

On every device (but the central syslogserver or you create a loop that fills the drive in mere seconds)

/etc/rsyslog.d/99-centralsyslog.conf

$PreserveFQDN on
*.*  @192.168.1.66

then on the central syslog server 192.168.1.66

/etc/rsyslog.d/01-syslog_receiver.conf

# provides UDP syslog reception
module(load="imudp")
input(type="imudp" port="514")

# provides TCP syslog reception
module(load="imtcp")
input(type="imtcp" port="514")

and also

/etc/rsyslog.d/20-save2postgresql.conf

# Load the PostgreSQL output module
module(load="ompgsql")

# Template for inserting logs
template(name="pgsql-template" option.sql="on" type="string" string="INSERT INTO system_events (hostname, facility, priority, tag, message) VALUES ('%HOSTNAME%', %syslogfacility%, %syslogpriority%, '%syslogtag%', '%msg%' )") 

# Send logs to PostgreSQL
*.emerg    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.panic    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.alert    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.crit     :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.error    :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
*.err      :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.warning  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.warn     :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.notice  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.info  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template
#*.debug  :ompgsql:127.0.0.1,syslog,syslog_user,WeakPassword;pgsql-template

Make sure you install postgres, the rsyslog-psql module and create the database and tables.

Grafana can run on the same or any other server.