Tag: nagios

Black Rock City Wifi Summit 2015

The Black Rock City Wifi Summit 2015 took place yesterday. It was an interesting discussion with a collection of people from inside the Burning Man organization and artists / community who want to provide wifi to their art installation and camps. It was held at the Internet Archive which is a very interesting idea / place.

Tidbits

There are some bits of information that I learned or re-learned while at the summit.

BRC internet basically uses all the available bandwidth coming into Gerlach, around 40Mb/s, to support all the services for a temporary city of 70,000 people.

In 2014, the bandwidth was maxed out on Thursday before the event started.

Internet access is becoming / has become critical for the support of Burning Man. Without it, the event might not be able to take place.

Ubiquiti AirMax protocol deals well with the hidden node problem.

It’s important that the uplink antenna be stable and pointed at the center camp. Movement or misalignment of the antenna causes problems for everyone in that sector by reducing the available bandwidth.

The uplink antenna needs line of sight to the center camp tower.

If you need to use tape to seal anything, use aluminized hvac tape instead of duct tape. The glue on duct tape deteriorates rapidly in the heat and dust of the playa. The aluminized tape will last much longer.

During the weeks before and the event, download is maxed out. The week after, it’s the upload bandwidth that’s maxed out.

At 4:00 and A there’s usually a phone booth that uses VOIP to allow calls to the outside world. Calls are limited to 3 minutes. Because of bandwidth usage, the other side will be able to hear you while you might not be able to hear them. That makes for interesting conversations when you’re stoned.

Don’t use 5Ghz band for access points or other projects. It’s used for the uplinks and putting up local access points (or anything else) in the 5Ghz band will just interfere with the uplinks.

There’s lots of local, on playa bandwidth available. If there’s lots of data that you

Mount the local access points 6 or 7 feet in the air, no higher. This will help reduce interference and the hidden node problem on the local wifi connections.

If there’s any data you want to make available, consider setting up a server on the playa. If you do, let the folks on the BRC wifi mailing lists know about it. They may be able to help let people know about the server by including it on the captive portal page, setting up a dedicated IP address and / or a DNS entry.

Protect the access point and uplink power sources from water and dust. A simple way to do this is to put them in a tupperware type container. Cut a notch near the edge to allow any wires into the container and seal it closed.

There are no guarantees. The bandwidth available is completely subjective to the current needs and wifi in camps and art installations is at the bottom of the priority list.

Lastly and perhaps most importantly, the folks running the BRC network are at Burning Man to enjoy themselves. Leave them alone and talk with other camps and installations about any issues that you may be experiencing.

Setup On The Playa

There are two main components needed to setup wifi on the playa – an uplink radio and an access point. The uplink radio runs in the 5Ghz band and the access point in the 2.4Ghz band. The recommended uplink radio is the NanoBeam from Ubiquiti. A NanoBridge will work if you have a spare one laying about. The recommended access point is a Ubiquiti Unifi AP.

To start, reset the NanoBeam to factory settings before bringing to the playa can help you save some hassles and wasted time on playa. With it reset to factory defaults, mount it with a clear line of sight to the center camp tower and power it up. It should connect and start provisioning. This can take a while, as in hours depending on what else is going on. Use a laptop to monitor the setup and configuration.

After the NanoBeam has connected, been provisioned and received it’s IP address, connect and power up the local access point.

Consider a Unifi AP Outdoor. The rain last year helped several people discover that the Unifi AP’s aren’t waterproof. Water and playa dust will do corrosive wonders to exposed connections.

Note that the UniFi AP Outdoor 5G runs in the 5Ghz band, don’t use it.. Why? Because 5Ghz is used for the uplinks.

The configuration of the Unifi AP’s that’s downloaded from the playa NOC will most likely include a captive portal that times out after some period. This will help manage the bandwidth needed since you’ll have to re-accept the terms and conditions page to connect back to the Internet.

If you don’t use a Unifi AP, set your access point SSID to the camp name / location and make it discoverable. Set up a captive portal for your users. It’ll help manage available bandwidth and provide service to the community:

Projects

I’ve been toying with different ideas for project on the playa of which two seem to be most likely to happen this year.

Wifi Kiosks

The first is a local wifi spot and map server but since there will be playamap onsite and there’s lots of local bandwidth available, I think the mapping may get put on the back burner for this year.

It turns out that there were a few like minded people at the summit talking about projects to bring wifi to the masses. I’m planning to work with the folks at http://ki7wv.net/ to create some solar powered wifi kiosks and distribute them on the playa.

Trasher

Set up a Raspberry Pi image that pulls SNMP status from the Ubiquity NanoBeam uplinks. This will allow an uplink operator to get an idea on how well their connections works over time.

I’m currently thinking this can be done with Cacti or Nagios or perhaps just a simple Graphite setup. I think Nagios is way overkill for this but we’ll see. Some experimentation needs to be done…

Tags : , ,

Nagios Monitoring of Directories

I had a recent occurrence at work that caused me to look around for a tool to monitor a directory for any changes made. Since there didn’t seem to be anything out there, I created a check called dirchanged. It looks at all the files in a directory and creates an sha256 has of the names and contents of the files. That hash is compared to a known value to determine if there have been any changes made.

There are a couple of issues with this check specifically that it doesn’t look into subdirectories and that the hash for comparison is passed on the command line from within the Nagios configuration files. I think the first issue will be fixed soon enough w/ a flag to indicate if the directory tree is to be traversed. The second issue is more cumbersome in that the hash value has to be stored somewhere. I’m not yet certain that putting it in the Nagios configuration files is better than putting it somewhere on the target file system. From the security standpoint, having the check not stored on the target file system is better, much less chance of it being changed by bad guys.

I’ll let it run for a while and see how it behaves and if changes are warranted.

Tags : ,

Web service monitoring w/ Nagios and JSON

I’m using Nagios to act as a watch dog for my network and the various services that live on it. Nagios does the job pretty well. It lets me know when there’s a problem, when things are back to normal and generally keeps on eye on things for me.

The checks that Nagios performs are done through a series of check commands. These commands are your typical Unix style program with the exceptions that they produce a single line of text that describes the state of the item being checked and the exit value let’s Nagios know what’s going on.

So for instance, to check the health of the web service on the localhost:

peter@sybil:~$ /usr/lib/nagios/plugins/check_http -H localhost
HTTP OK HTTP/1.1 200 OK - 361 bytes in 0.001 seconds |time=0.001021s;;;0.000000 size=361B;;;0
peter@sybil:~$ echo $?
2
peter@sybil:~$

The single line of text that is displayed follows a specific format. It starts with the prefix of what’s being tested, HTTP. Next is the status, OK. This can be OK, WARNING, CRITICAL or UNKNOWN. Everything after the status is eye candy that provide details that are specific to the test being done. Nagios doesn’t really care about it but it does provide important details when looking at problems that may be occurring.

Writing these check program in Python is pretty straight forward.

I recently had a situation where our ISP moved our web servers from one physical machine to another. This resulted in the credit cards processing for our online store to fail. The payment provider uses the IP address of the server as part of the authentication process when submitting credit cards for processing. Since the server changed, the IP address changed. Things went around in circles for a while until we figured out the problem and gave the new IP address to the payment
provider.

I thought is would be a good additional Nagios check for the store web site to check on the IP address of the physical server. Unfortunately, the ISP doesn’t provide access to the IP address. But they do provide access to the hostname.

To get the hostname, I added a simple CGI program that determines the hostname and then packages it up as a JSON data structure.

#!/usr/bin/env python
 
"""
Bundle the hostname up as a JSON data structure.
 
Copyright (c) 2009 Peter Kropf. All rights reserved.
"""
 
import cgi
import popen2
import sys
sys.path.insert(1, '/home/crucible/tools/lib/python2.4/site-packages')
sys.path.insert(1, '/home/crucible/tools/lib/python2.4/site-packages/simplejson-2.0.9-py2.4-linux-x86_64.egg')
 
import simplejson as json
 
field = cgi.FieldStorage()
print "Content-Type: application/json\n\n"
 
r, w, e = popen2.popen3('hostname')
host = r.readline()
r.close()
w.close()
e.close()
 
fields = {'hostname': host.split('n')[0]}
 
print json.dumps(fields)

There’s a couple of things to note. Since the ISP provides a very restrictive environment, I have to add the location of the simplejson module before it can be imported. It’s a bit annoying but it does work.

On the Nagios service side, I created a new check program called check_json. It takes the name of a field, the expected value and the URI from which to pull the JSON data.

#! /usr/bin/env python
 
"""
Nagios plugin to check a value returned from a uri in json format.
 
Copyright (c) 2009 Peter Kropf. All rights reserved.
 
Example:
 
Compare the "hostname" field in the json structure returned from
http://store.example.com/hostname.py against a known value.
 
    ./check_json hostname buenosaires http://store.example.com/hostname.py
"""
 
 
import urllib2
import simplejson
import sys
from optparse import OptionParser
 
prefix = 'JSON'
 
class nagios:
    ok       = (0, 'OK')
    warning  = (1, 'WARNING')
    critical = (2, 'CRITICAL')
    unknown  = (3, 'UNKNOWN')
 
 
def exit(status, message):
    print prefix + ' ' + status[1] + ' - ' + message
    sys.exit(status[0])
 
 
parser = OptionParser(usage='usage: %prog field_name expected_value uri')
options, args = parser.parse_args()
 
 
if len(sys.argv) < 3:
    exit(nagios.unknown, 'missing command line arguments')
 
 
field = args[0]
value = args[1]
uri = args[2]
 
try:
    j = simplejson.load(urllib2.urlopen(uri))
except urllib2.HTTPError, ex:
    exit(nagios.unknown, 'invalid uri')
 
if field not in j:
    exit(nagios.unknown, 'field: ' + field + ' not present')
 
if j[field] != value:
    exit(nagios.critical, j[field] + ' != ' + value)
 
exit(nagios.ok, j[field] + ' == ' + value)

Some checking is done to ensure that the JSON data can be retrieved, that the needed field is in the data and then that the field’s value matches what’s expected.

These examples show the basic testing that’s done and the return values:

peter@sybil:~$ /usr/lib/nagios/plugins/check_json hostname buenosaires http://store.thecrucible.org/hostname.py
JSON OK - buenosaires == buenosaires
peter@sybil:~$ echo $?
0
peter@sybil:~$ /usr/lib/nagios/plugins/check_json hostname buenosaires http://store.thecrucible.org/hostname.p
JSON UNKNOWN - invalid uri
peter@sybil:~$ echo $?
3
peter@sybil:~$ /usr/lib/nagios/plugins/check_json hostname buenosairs http://store.thecrucible.org/hostname.py
JSON CRITICAL - buenosaires != buenosairs
peter@sybil:~$ echo $?
2
peter@sybil:~$ /usr/lib/nagios/plugins/check_json ostname buenosaires http://store.thecrucible.org/hostname.py
JSON UNKNOWN - field: ostname not present
peter@sybil:~$ echo $?
3
peter@sybil:~$

Once the Nagios server is configured with the new command, the hostname on the server can be monitored and hopefully ease any problems that may occur then next time things change at the ISP.

More details on Nagios can be found at http://nagios.org and on developing check program at http://nagiosplug.sourceforge.net/developer-guidelines.html.

Tags : ,