# Monitoring::Sneck

## SYNOPSIS

```
sneck -u [-C <cache file>] [-f <config file>] [-p] [-i] [-d] [-q] [-l] [-r]

sneck -c [-C <cache file>] [-b]

sneck [-f <config file>] [-p] [-i] [-r]

sneck -t [-f <config file>]
```

## FLAGS

### -f config_file

The config file to use.

Files ending in .yaml or .yml are read as YAML, which needs YAML::XS.

Default :: /usr/local/etc/sneck.conf

### -p

Pretty it in a nicely formatted format.

### -C cache_file

The cache file to use.

Default :: /var/cache/sneck.cache

A secondary cache file based on this name is also created. By default
it is /var/cache/sneck.cache.snmp and always holds the GZip+BASE64
compressed version.

### -u

Update the cache file. Will also print the was written to it.

### -c

Print the cache file. Please note that -p or -i won't affect
this as this flag only reads/prints the cache file.

### -b

When used with -c, print the LibreNMS style GZip+BASE64 compressed
cache instead.

### -i

Includes the config file used.

### -d

Print debugging info if called with -u.

### -q

Don't print the results for -u. Exit quietly.

### -l

Enable locking for -u so more than one instance can't run at a time.

### -r

Run any restarts whose checks failed. Without this, restarts are only
reported as `restarts disabled`. Meant to be used with -u from cron.
Never use it from snmpd.

Restart state, used for `min_interval` and `max_retries`, is kept in the
cache file name with `.restarts` added, so by default
/var/cache/sneck.cache.restarts.

Use -l with this to make sure two runs can't restart things at the same
time.

### -t

Test the config file. Prints any errors and warnings, then exits 0 if
there are no errors and 1 if there are or the file can't be read.
Warnings, such as a check using a undefined variable, do not affect the
exit code.

## CONFIG FORMAT

The sneck format is described below. Files ending in .yaml or .yml
use the YAML format instead, described under YAML CONFIG.

Each line has leading spaces and tabs removed before it is looked at. A
trailing \r is also removed, so files with CRLF line endings work.

Blank lines are ignored.

Lines starting with # are comments and are ignored.

- `env NAME=value` :: Sets a environment variable. The `env` is case
  insensitive. The value may be empty. These may be set more than once,
  with the last one winning. They are only applied if the whole config
  is valid.

- `NAME=value` :: A variable. The name is before the first =, the value
  is everything after it. The value may be empty.

- `name|command` :: A check. The command is everything after the first
  | with leading whitespace removed. It may not be empty.

- `%name|command` :: A debug check. Same as a check, but not counted
  towards any of the counts. It exists purely for debugging. The leading
  % is not part of the name, so a check and a debug check may share a
  name.

- `@name|options|command` :: A restart. Options are space separated
  `key=value`, with `checks` and `depends` being comma separated lists.
  Values containing spaces may be quoted with `"` or `'`. See RESTARTS.

Names are made up of `A-Z`, `a-z`, `0-9`, and `_`.

Any other sort of line is an error. Every bad line is reported, along
with its line number.

Variables are used in commands in the form `%NAME%`. A reference to a
variable that is not defined is left as written and produces a warning,
as it may just be part of the command, such as `date +%Y%m%d`. Use `-t`
to see warnings.

Variable, check, debug check, and restart names may not be redefined.

## EXAMPLE CONFIG

```
env PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
# this is a comment
GEOM_DEV=foo
geom_foo|/usr/local/libexec/nagios/check_geom mirror %GEOM_DEV%
does_not_exist|/bin/this_will_error yup... that it will
    
does_not_exist_2|/usr/bin/env /bin/this_will_also_error

#includes route info
%routes|netstat -rn
```

The first line sets the %ENV variable PATH.

The second is ignored as it is a comment.

The third sets the variable GEOM_DEV to 'foo'

The fourth creates a check named geom_foo that calls check_geom_mirror
with the variable supplied to it being the value specified by the
variable GEOM_DEV.

The fith is a example of an error that will show what will happen when
you call to a file that does not exit.

The sixth line will be ignored as it is blank.

The seventh is a example of another command erroring.

When you run it, you will notice that errors for lines 4 and 5 are
printed to STDERR. For this reason you should use '2> /dev/null' when
calling it from snmpd or '2> /dev/null > /dev/null' when calling from
cron. 

## YAML CONFIG

YAML configs need YAML::XS, which is optional and only loaded when a
YAML config is used.

```
env:
  PATH: /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin
vars:
  GEOM_DEV: foo
checks:
  geom_foo: /usr/local/libexec/nagios/check_geom mirror %GEOM_DEV%
  does_not_exist: /bin/this_will_error yup... that it will
debugs:
  routes: netstat -rn
```

- `env` :: Environment variables to set, in sorted name order. Only
  applied if the whole config is valid.

- `vars` :: Variables.

- `checks` :: Checks, with the command as the value.

- `debugs` :: Debug checks, with the command as the value.

- `restarts` :: Restarts, each a mapping of options plus `command`.
  See RESTARTS.

Any other top level key is an error. Any section may be left out or
empty.

Names, variables, and undefined variable warnings work the same as the
sneck format. Errors and warnings point to a path, such as
`checks.geom_foo`, as YAML::XS does not give line numbers.

Values must be strings or numbers. An empty value is an empty string
for `env` and `vars` and an error for `checks` and `debugs`.

Some things to watch for.

- Unquoted `true` becomes 1 and `false` becomes an empty string. Quote
  values like those.

- A value starting with `%` must be quoted, as `%` can't start a plain
  YAML value.

- A value containing `: ` or ending in `:`, such as `-c 1:`, must be
  quoted, or it is read as a mapping.

- Duplicate keys are not caught. YAML::XS silently keeps the last one.

## RESTARTS

A restart is a command run when enough of the checks it watches fail.
They only run with -r. Otherwise they are just reported.

```
http_check|/usr/local/libexec/nagios/check_http -H localhost
php_check|/usr/local/libexec/nagios/check_procs -C php-fpm -c 1:
@php_fpm|checks=php_check|/usr/sbin/service php_fpm restart
@httpd|checks=http_check,php_check threshold=2 depends=php_fpm cascade=1 timeout=60|/usr/sbin/service apache24 restart
```

```
restarts:
  php_fpm:
    command: /usr/sbin/service php_fpm restart
    checks: [php_check]
  httpd:
    command: /usr/sbin/service apache24 restart
    checks: [http_check, php_check]
    threshold: 2
    depends: [php_fpm]
    cascade: true
    timeout: 60
```

- `checks` :: Required. The checks to watch. Debug checks can't be
  watched.

- `threshold` :: How many of the watched checks must fail for it to
  trigger. Default :: 1

- `depends` :: Other restarts this one depends on. When both trigger,
  the ones depended on run first. A depend that didn't trigger is
  assumed fine and isn't run. If a depend runs and fails, this one is
  skipped. Cycles are errors.

- `cascade` :: If 1, this also runs when any of its depends ran without
  failing, even if its own threshold wasn't met. Default :: 0

- `ignore_unknown` :: If 0, unknown counts as failed. Default :: 1

- `ignore_errored` :: If 0, errored counts as failed. This is any exit
  other than 0 to 3, dying on a signal, or not being able to run the
  check. Default :: 1

- `min_interval` :: Minimum seconds between runs. 0 turns it off.
  Default :: 180

- `max_retries` :: How many times in a row it will run for its threshold
  before giving up until its checks recover. 0 means always retry. Runs
  from cascade don't count. Default :: 0

- `timeout` :: Seconds to wait on the command before giving up on it.
  On timeout its output pipes are closed and it is left running. Nothing
  is sent to it, but if it writes again it gets SIGPIPE, or whatever it
  does when the reader goes away. A timeout counts as failed, with an
  exit of -1. Default :: 30

- `not_every` :: A maintenance window, as a five field cron spec. While
  local time matches it, the restart doesn't run. Numbers only, no names
  like `sat` or `jan`. Sunday is 0 or 7. If both day of month and day of
  week are given, either matching is enough, same as cron. Restarts
  depending on one held back by this act as if it didn't trigger. Must
  be quoted, such as `not_every="* 2-3 * * 0"`, or in YAML
  `not_every: '* 2-3 * * 0'`. Default :: none

Critical always counts as failed. Ok and warning never do.

A restart that ran and failed sets `.data.alert` and adds a line to
`.data.alertString`, along with any output from the command.

## USAGE

snmpd should be configured as below.

```
extend sneck /usr/bin/env PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin /usr/local/bin/sneck -c
```

Then just setup a entry in like cron such as below.

```
*/5 * * * * /usr/bin/env PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin  /usr/local/bin/sneck -u 2> /dev/null > /dev/null
```

Most likely want to run it once per polling interval.

You can use it in a non-cached manner with out cron, but this will result in a
longer polling time for LibreNMS or the like when it queries it.

## RETURN HASH/JSON

The generated JSON/hash is as below in jpath notation.

- .data.alert :: 0/1 boolean for if there is a aloert or not.

- .data.ok :: Count of the number of ok checks.

- .data.warning :: Count of the number of warning checks.

- .data.critical :: Count of the number of critical checks.

- .data.unknown :: Count of the number of unkown checks.

- .data.errored :: Count of the number of errored checks.

- .data.alertString :: The cumulative outputs of anything that
  returned a warning, critical, or unknown.

- .data.vars :: A hash with the variables to use.

- .data.time :: Time since epoch.

- .data.time :: The hostname the check was ran on.

- .data.config :: The raw config file if told to include it.

- $hash{data}{run_time} :: How long it took to run all checks.

- .data.restarted :: Count of the number of restarts ran.

- .data.restart_state_error :: Only present if the restart state file
  could not be read or written.

For the following `$name` is the name of the check ran.

- .data.checks.$name :: A hash with info on the checks ran.

- .data.checks.$name.check :: The command pre-variable substitution.

- .data.checks.$name.ran :: The command ran.

- .data.checks.$name.output :: The output of the check.

- .data.checks.$name.exit :: The exit code.

- .data.checks.$name.error :: Only present it died on a signal or
  could not be executed. Provides a brief description.

- $hash{data}{checks}{$name}{run_time} :: How long it took to run the checks.

For the following `$name` is the name of the debug check ran.

- .data.debugs.$name :: A hash with info on the checks ran.

- .data.debugs.$name.check :: The command pre-variable substitution.

- .data.debugs.$name.ran :: The command ran.

- .data.debugs.$name.output :: The output of the check.

- .data.debugs.$name.exit :: The exit code.

- .data.debugs.$name.error :: Only present it died on a signal or
  could not be executed. Provides a brief description.

 - $hash{data}{checks}{$name}{run_time} :: How long it took to run the debug.

For the following `$name` is the name of the restart. Every restart has
an entry, even when restarts are disabled.

- .data.restarts.$name.triggered :: 0/1 for if its threshold was met.

- .data.restarts.$name.ran :: 0/1 for if it ran.

- .data.restarts.$name.reason :: Why it did or didn't run. One of
  `threshold`, `cascade from $depend`, `cooldown, $N seconds left`,
  `max retries reached`, `skipped, dependency $depend failed`,
  `maintenance window`, `not triggered`, or `restarts disabled`.

- .data.restarts.$name.failed_checks :: The watched checks that failed.

- .data.restarts.$name.threshold :: The threshold.

- .data.restarts.$name.attempts :: Runs for its threshold since its
  checks last recovered.

- .data.restarts.$name.command :: The command pre-variable substitution.

- .data.restarts.$name.ran_command :: The command ran. Only if it ran.

- .data.restarts.$name.output :: The output. Only if it ran.

- .data.restarts.$name.exit :: The exit code, or -1 if it timed out. Only if
  it ran.

- .data.restarts.$name.error :: Only present if it timed out, died on a
  signal, or could not be executed.

- .data.restarts.$name.run_time :: How long it took. Only if it ran.

## INSTALLING

### FreeBSD

```
pkg install p5-JSON p5-JSON-XS p5-File-Slurp p5-Proc-PID-File p5-DateTime-Event-Cron p5-App-cpanminus
cpanminus Monitoring::Sneck
```

### Debian

```
apt-get install libjson-perl libjson-xs-perl libfile-slurp-perl libproc-pid-file-perl libdatetime-event-cron-perl cpanminus
cpanminus Monitoring::Sneck
```

### YAML Support

For YAML configs, also install YAML::XS.

```
# FreeBSD
pkg install p5-YAML-LibYAML

# Debian
apt-get install libyaml-libyaml-perl
```

### From Src

```
perl Makefile.PL
make
make test
make install
```