Jun 242014
 

As many of you know from my earlier post: http://cornempire.net/2012/02/26/tournament-bracket-display/ I’m always trying to display brackets online. While my previous solution was ok, it isn’t the easiest to use.

So, I’ve created a few Google Sheets templates which can be used for displaying tournament brackets. I’ve created 4, 8, 16 and 32 player brackets. You just fill in the names, and the scores, and you are all set. The nice part of this solution is that you can use the Google Drive app on your phone/tablet to update in real time, and of course you don’t need any server infrastructure to manage this solution.

You should be able to view these, and make a copy into your Google account. Then create your own brackets based on the file. If there are any problems, let me know.

Here are the templates:
4 Player
8 Player
16 Player
32 Player

Here is an example of a completed bracket: https://docs.google.com/spreadsheets/d/1FyXqWXpNE0z9Fx419H0WI1ZRoIvTXXyyE9Z5GFzd_7U/edit?usp=sharing

Jan 212014
 

I just recently remodified an install of awstats to detect ios and iphone, and while I was at it, I reapplied my changes to detect our xymon server as well.

Following the instructions here: http://cornempire.net/2009/08/31/adding-a-robot-to-awstats/ and here: http://wiki.cornempire.net/awstats/awstatsrobots still work. I just made a small tweak to match our new version of xymon, and also to make the detection more broad, which should future proof it a bit: http://wiki.cornempire.net/awstats/awstatsrobots#updatejan_21_2014

As a side note, if you are trying to detect ios for iPhones and iPads, this tutorial is useful: http://serverfault.com/questions/248692/analyze-http-logs-looking-for-ios except for one thing.

I received an error about a mismatch in the number of entries in the @OSSearchIDOrder and %OSHashID. This was because more recent versions of awstats includes basic ios detection, and because the footprint of the new code has the same ios label as the stock code, it would throw this error. Commenting out the built in ios detection did the trick for me. I may try to expand upon the ios section a bit and try to detect version numbers in a future version. I may also create a patch that breaks out iOS and Android devices to a new family called tablets, or mobile devices….maybe.

Dec 082013
 

I have updated my Hesk hack based on a request from a reader. You can view the new instructions or download the new files here: http://wiki.cornempire.net/hesk/hesk252

For those unaware, my previous post on this matter was here: http://cornempire.net/2010/07/31/hesk-enhancements-for-version-2-2/

2.5 introduced a lot of changes. One of the big ones that affected my hack was the ability to directly link to a category type. This is now built into the application, so I modified my hack to take advantage of this.

If you have any questions, let me know.

Dec 032013
 

At work we have started using a piece of software called Alfresco. We use the Share web interface for collaboration within our department. A co-worker and I attended the excellent Alfresco Summit: http://summit.alfresco.com/ in Boston. We learned a lot about the software and how extensible it is. It can be used in many ways, far more heavily then we are currently using it. We have hatched plans to use it for our departmental workflows, as well as building an online repository upon the deep repository backend that is available.

The application is written in Java and has lots of configuration and modification options. I’m going to highlight the first change we made to the Share UI here. I plan to document more in the future.

Goal: We plan to use our Share implementation mostly for internal collaboration and have it tied to our active directory for that purpose. However, we do plan to allow others to come in for some projects. Because of this we wanted to change the default option when creating a site from ‘Public’ to ‘Private’. This will help prevent staff from creating sites that are public by accident.

While at the conference, I spoke with an Alfresco Engineer and he explained how we could customize this page.

The file which controls the screen is called: create-site.get.html.ftl. I found it at ./tomcat/webapps/share/WEB-INF/classes/alfresco/site-webscripts/org/alfresco/modules/create-site.get.html.ftl

Grab a copy of all of the content in this file and make your changes.

Then, create the same file in this directory: ./tomcat/shared/classes/alfresco/web-extension/site-webscripts/org/alfresco/modules/ (you will likely have to create some of these directories yourself.) Note that the part of the original path after WEB-INF matches everything after the shared folder in the except for the addition of the web-extension folder in the override path. I’m sure that is relevant, which is why I’ll highlight it below:
Path 1: ./tomcat/webapps/share/WEB-INF/classes/alfresco/site-webscripts/org/alfresco/modules/
Path 2: ./tomcat/shared/classes/alfresco/web-extension/site-webscripts/org/alfresco/modules/

If you wanted to customize the HTML available in other parts of the app, you could probably drop the override file into the matching directory from its original location.

Anyway, once you have the file in place, you can restart Alfresco and it should be in play. This worked the first time I tried it, which was quite excellent!

Now, when you complete upgrades of your repository, you should back up your customizations and reapply them using the newer version of the code. The Alfresco team could change, add, or remove features from the page you are customizing and your override file will not take into consideration those changes. This could cause the behaviour to be erratic after the upgrade.

Aug 072013
 

Yesterday I harvested my first crop of celery from the green house. I started three heads of celery indoors after reading a post here: http://www.17apart.com/2012/02/growing-celery-indoors-never-buy-celery.html

Started as described in the article, in bowls of water in March or April of this year. About a month or so after starting, I transplanted into flower pots. I kept it mostly indoors for the spring on my kitchen window where it could get the afternoon sun. In May, I started putting the pots outside to give the celery more sunlight, and to give it some colder temperatures so that it could handle being outside. I’d bring them in each day so that the cold wouldn’t kill them.

Late May, early June, after our last snowfall, I planted outdoors in the greenhouse. Transplanting was pretty easy. I was surprised by how many roots had sprouted and wrapped around the flower pot. Here are a few pictures of the greenhouse a week or so after the transplant.

after-transplant

I decided to pull up one of the heads because there were a lot of brown stalks. I thought it was drying out, but after I picked it, it was clear that bugs had eaten some of the stalks. Looked like earwigs and they would get inside and hollow out the stalk. After cleaning the head, there were fewer ruined stalks then I thought, which was good.

Below are some pictures of the harvest:

harvest1

harvest2

Jun 212013
 

The Goal

Have all syslogs from all servers shipped to a central server so that we can query them in one spot, and review old logs in the event of a compromise using only free software.

After looking at a number of options, I settled on rsyslog for the server, standard syslog on the clients (for now), LogAnalyzer as the UI for the web, both mysql backend for rsyslog and file based backend. The configuration for rsyslog will be a little different then most of the tutorials out there. I wanted to be able to query across all servers at one time, and LogAnalyzer only allows you to configure specific endpoints. With most configuration examples you will find on the web, they show you how to either dump it all into a database (which will no doubt get huge if you don’t clean it up) or dump each server to a single file which rotates daily, which is not ideal for a LogAnalyzer end point because your config has to change daily. This solution will dump all events into a database for all servers which will be configured as one realtime endpoint. rsyslog will also be configured to dump to a file, and this file will be rotated monthly using the usual logrotate scripts. Several archival (non-compressed) files will also be configured in LogAnalyzer for historical purposes.

Install rsyslog

I’m not going to go into this in detail. I’m going to assume you can already do this. But here are some useful notes:

  • My solution was built on RHEL5 using the packages available from yum. You will need the rsyslog package, as well as apache, php, and mysql for the database. I used link 2 below for this, although I didn’t remove any of the previous syslog packages. I just turned them off. Additionally, you only need to install rsyslog on your central server for this solution, not on each client.
  • I used this template line at the end of my rsyslog config: $template DailyPerHostLogs,”/var/log/LOGHOSTS/%HOSTNAME%/%HOSTNAME%.log”

Configure DB

I’m going to assume you have already installed the DB, but if not, here is a cheat sheet:

  • yum install mysql-server

Run this command (or create the database in another way):

  • mysql -uroot -p < /usr/share/doc/rsyslog-mysql-3.22.1/createDB.sql

Then create a user and password that can access the database from the host the server is on.

Configure rsyslog to log to both

Now this is where the magic happens. Provided you took some part of a sample config from somewhere, you should have something like this is your rsyslog:

$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imudp.so
#load the network stuff
$UDPServerAddress 0.0.0.0
$UDPServerRun 514
#reduce any duplicates
$RepeatedMsgReduction on
# The template that wil format the message as it is writen to the file
# you can edit this line if you want to customize te message format
$template TraditionalFormat,"%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
#*.info;mail.none;authpriv.none;cron.none                /var/log/messages
if \
        $source == 'YOUR.CENTRAL.SERVER' \
        and \
               $syslogseverity <= '6' \
        and ( \
                        $syslogfacility-text != 'mail' \
                and \
                        $syslogfacility-text != 'authpriv' \
                and \
                        $syslogfacility-text != 'cron' \
        ) \
then   /var/log/messages;TraditionalFormat
#authpriv.* /var/log/secure;TraditionalFormat
# The authpriv file has restricted access.
#authpriv.*                                              /var/log/secure
if \
        $source == 'YOUR.CENTRAL.SERVER' \
                and \
        $syslogfacility-text == 'authpriv' \
then    /var/log/secure;TraditionalFormat
# Log anything (except mail) of level info or higher.
# Don't log private authentication messages!
# mail.* /var/log/maillog;TraditionalFormat
if \
        $source == 'YOUR.CENTRAL.SERVER' \
                and \
        $syslogfacility-text == 'mail' \
then    /var/log/maillog;TraditionalFormat
 
# Log cron stuff
#cron.* /var/log/cron;TraditionalFormat
if \
        $source == 'YOUR.CENTRAL.SERVER' \
                and \
        $syslogfacility-text == 'cron' \
then    /var/log/cron;TraditionalFormat
# Everybody gets emergency messages
#*.emerg *
if \
        $source == 'YOUR.CENTRAL.SERVER' \
                and \
        $syslogseverity-text == 'emerg' \
then    *
# this line creates a template that will store the messages for each host in a seperate file.
# The same file will always be used, and should be rotated using a Log Rotate script.
$template DailyPerHostLogs,"/var/log/LOGHOSTS/%HOSTNAME%/%HOSTNAME%.log"
*.* -?DailyPerHostLogs;TraditionalFormat

What we need to add for the mysql logging is:

$ModLoad ommysql.so
*.*  :ommysql:localhost OR IP OF MYSQL SERVER,Syslog,USERNAME,PASSWORD

I added the above code to the very end of the file. Then I restarted rsyslog.

Then I configured a new entry in my config file for LogAnaylzer. If you haven’t already installed it yet, you can use the install wizard to set this up. But if you have installed it, you will need to use the config.php file to make changes. The sample included in the comments seemed a little off, so this is what I used:

$CFG['Sources']['Source2']['ID'] = "Source2";
$CFG['Sources']['Source2']['Name'] = "All Servers Realtime";
$CFG['Sources']['Source2']['Description'] = "";
$CFG['Sources']['Source2']['SourceType'] = SOURCE_DB;
$CFG['Sources']['Source2']['MsgParserList'] = "";
$CFG['Sources']['Source2']['DBTableType'] = "winsyslog";
$CFG['Sources']['Source2']['DBType'] = DB_MYSQL;
$CFG['Sources']['Source2']['DBServer'] = "localhost OR YOUR DB HOST";
$CFG['Sources']['Source2']['DBName'] = "Syslog";
$CFG['Sources']['Source2']['DBUser'] = "YOUR DB USERNAME";
$CFG['Sources']['Source2']['DBPassword'] = "YOUR DB PASSWORD";
$CFG['Sources']['Source2']['DBTableName'] = "SystemEvents";
$CFG['Sources']['Source2']['ViewID'] = "SYSLOG";

You can optionally configure a mysql database to store the configuration settings and manage user accounts. I eventually did this for our install, and it did make configuration much simpler.

Log Rotate and mysql Purge Script

It is a good idea to rotate the logs regularly and update the database to purge older entries.

Place this file in /etc/logrotate.d/ and call it rsyslog:

# Rotate each log monthly
/var/log/LOGHOSTS/*/*.log {
    monthly
    sharedscripts
    nocompress
#    extension .1
    rotate 4
    create 644 root root
    postrotate
         restart rsyslog >/dev/null 2>&1 || true
    endscript
}
#/var/log/LOGHOSTS/*/*.log.1 {
#    monthly
#    extension .2
#}
#/var/log/LOGHOSTS/*/*.log.2 {
#    monthly
#    extension .3
#}
/var/log/LOGHOSTS/*/*.log.3 {
    rotate 8
    monthly
    delaycompress
    maxage 365
}

EDIT: My log rotate script didn’t work as expected. Try this instead. Another option could be to reverse the order of the files above, handle .3 first, then .2…etc It seemed to work ok for the first month, then started added extra .# extensions and messed things up a bit.

To purge the database and keep your logs clean, just set up this script to run as a cronjob:

#!/bin/sh
# Special account to run this query only.
#MYSQL queries
RECORDSTODELETE=`mysql -u{MYSQL USERNAME} -p{MYSQL PASSWORD} --silent -e "SELECT COUNT(*) FROM Syslog.SystemEvents WHERE Syslog.SystemEvents.ReceivedAt &lt; DATE_SUB(NOW(), INTERVAL 30 DAY);&quot;`
RESULTS=`mysql -u{MYSQL USERNAME} -p{MYSQL PASSWORD} --silent -e &quot;use Syslog; DELETE from Syslog.SystemEvents WHERE Syslog.SystemEvents.ReceivedAt &lt; DATE_SUB(NOW(), INTERVAL 30 day);&quot;`
`logger -i Purged ${RECORDSTODELETE} records from logging database`

Odds and Ends

If you got this all to work, you may notice that your database based sources show more information than your file based sources. If you want facility and severity to display for text based entries, you will need to setup rsyslog to include this information in the file.

First, I’m using version 3.22.1 of rsyslog from the RHEL5 repos. Which is very old. I used this template to change what is logged to the file:

$template TraditionalFormatWithPRI,"%syslogfacility%.%syslogpriority%: %timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n"

I also added a new database template that separates the process ID from the syslog tag:

$template SQLWithProcessID,"insert into SystemEvents (Message, Facility, FromHost, Priority, DeviceReportedTime, ReceivedAt, InfoUnitID, SysLogTag, ProcessID) values ('%msg%', %syslogfacility%, '%HOSTNAME%', %syslogpriority%, '%timereported:::date-mysql%', '%timegenerated:::date-mysql%', %iut%, '%syslogtag:R,ERE,1,FIELD:([a-zA-Z\/]+)(\[[0-9]{1,5}\])*:--end%', '%syslogtag:R,ERE,1,BLANK:\[([0-9]{1,5})\]--end%')",sql

and then make a slight change to our database logging line:

*.*  :ommysql:localhost OR IP OF MYSQL SERVER,Syslog,USERNAME,PASSWORD;SQLWithProcessID

But you then have to change LogAnalyzer in order to read the extra data.

In the $APPROOT$/classes/logstreamlineparsersyslog.class.php your first four entries in the if block should look like this:

                // Standard syslog with process and facility/severity. TGH Apr. 4, 2013
                if ( preg_match("/(.*)\.(.*): (...)(?:.|..)([0-9]{1,2} [0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}) ([a-zA-Z0-9_\-\.]{1,256}) ([A-Za-z0-9_\-\/\.]{1,32})\[(.*?)\]:(.*?)$/", $szLine, $out ) )
                {
                        // Copy parsed properties!
                        $arrArguments[SYSLOG_FACILITY] = strtoupper($out[1]);
                        $arrArguments[SYSLOG_SEVERITY] = strtoupper($out[2]);
                        $arrArguments[SYSLOG_DATE] = GetEventTime($out[3] . " " . $out[4]);
                        $arrArguments[SYSLOG_HOST] = $out[5];
                        $arrArguments[SYSLOG_SYSLOGTAG] = $out[6];
                        $arrArguments[SYSLOG_PROCESSID] = $out[7];
                        $arrArguments[SYSLOG_MESSAGE] = $out[8];
                }
                // Standard syslog with process.
                elseif ( preg_match("/(...)(?:.|..)([0-9]{1,2} [0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}) ([a-zA-Z0-9_\-\.]{1,256}) ([A-Za-z0-9_\-\/\.]{1,32})\[(.*?)\]:(.*?)$/", $szLine, $out ) )
                {
                        // Copy parsed properties!
                        $arrArguments[SYSLOG_DATE] = GetEventTime($out[1] . " " . $out[2]);
                        $arrArguments[SYSLOG_HOST] = $out[3];
                        $arrArguments[SYSLOG_SYSLOGTAG] = $out[4];
                        $arrArguments[SYSLOG_PROCESSID] = $out[5];
                        $arrArguments[SYSLOG_MESSAGE] = $out[6];
                }
                // Sample (Syslog): syslog with facility.priority and no process - TGH Apr. 4, 2013
                else if ( preg_match("/(.*)\.(.*): (...)(?:.|..)([0-9]{1,2} [0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}) ([a-zA-Z0-9_\-\.]{1,256}) ([A-Za-z0-9_\-\/\.]{1,32}):(.*?)$/", $szLine, $out ) )
                {
                        // Copy parsed properties!
                        $arrArguments[SYSLOG_FACILITY] = strtoupper($out[1]);
                        $arrArguments[SYSLOG_SEVERITY] = strtoupper($out[2]);
                        $arrArguments[SYSLOG_DATE] = GetEventTime($out[3] . " " . $out[4]);
                        $arrArguments[SYSLOG_HOST] = $out[5];
                        $arrArguments[SYSLOG_SYSLOGTAG] = $out[6];
                        $arrArguments[SYSLOG_MESSAGE] = $out[7];
                }

                // Sample (Syslog): Mar 10 14:45:39 debandre syslogd 1.4.1#18: restart
                else if ( preg_match("/(...)(?:.|..)([0-9]{1,2} [0-9]{1,2}:[0-9]{1,2}:[0-9]{1,2}) ([a-zA-Z0-9_\-\.]{1,256}) ([A-Za-z0-9_\-\/\.]{1,32}):(.*?)$/", $szLine, $out ) )
                {
                        // Copy parsed properties!
                        $arrArguments[SYSLOG_DATE] = GetEventTime($out[1] . " " . $out[2]);
                        $arrArguments[SYSLOG_HOST] = $out[3];
                        $arrArguments[SYSLOG_SYSLOGTAG] = $out[4];
                        $arrArguments[SYSLOG_MESSAGE] = $out[5];
                }

Once you put it all together, you should have solid log viewer for all of your servers at once, as well as access to historical logs.

Tutorials

Here are some helpful tutorials that I found useful:

  1. Install docs for LogAnalyzer: http://loganalyzer.adiscon.com/doc/install.html
  2. RHEL Install and config of rsyslog: http://www.howtoforge.com/building-a-central-loghost-on-centos-and-rhel-5-with-rsyslog
  3. rsyslog and mysql: http://www.dawnstar.id.au/linux-documentation/configuring-rsyslog-log-mysql-rhel-52/
Jun 052013
 

Recently had a problem where awstats stopped processing log entries from one of my sites while the others worked fine. While some of these items don’t make sense to check in this instance, you may want to give them a look if you are having problems.

Check a few things first:

  1. Do you have enough disk space? – Yes
  2. Is the log file getting updated? (I was shipping logs from one server to another for processing) – Yes
  3. Has the log format changed? (Take a look at historical logs if available to make sure. Also check your /etc/awstats/awstats.config.conf file to make sure it is the same here) – All good
  4. Files still getting written to your data dir? Do you have any contents in any missing files? (/var/lib/awstats/awstats{month}{year}{day}.txt) – New files are there for working sites, nothing present for the failing site.
  5. Turn on showing dropped records and what did it say? – Dropped record (method/protocol ‘rtmp’ not qualified when LogType=S):
  6. Have you made any changes to the awstats application recently? – Yep….wait what?

It seems at some point we upgraded awstats to the 7.0 branch, and had previously made customizations to the /usr/local/awstats/wwwroot/cgi-bin/awstats.pl to handle our FMS logs. We did it according to the instructions here: http://www.wowza.com/forums/showthread.php?163-Log-Analysis

When we did the upgrade, we overwrote our changes, an this one log stopped processing.

So, I made the following change to /usr/local/awstats/wwwroot/cgi-bin/awstats.pl around line 18158 to make this block:

                        ( $LogType eq 'W' || $LogType eq 'S' )
                        && (   uc($field[$pos_method]) eq 'GET'
                                || uc($field[$pos_method]) eq 'MMS'
                                || uc($field[$pos_method]) eq 'RTSP'
                                || uc($field[$pos_method]) eq 'HTTP'
                                || uc($field[$pos_method]) eq 'RTP' )

look like this:

                        ( $LogType eq 'W' || $LogType eq 'S' )
                        && (   uc($field[$pos_method]) eq 'GET'
                                || uc($field[$pos_method]) eq 'MMS'
                                || uc($field[$pos_method]) eq 'RTSP'
                                || uc($field[$pos_method]) eq 'RTMP'
                                || uc($field[$pos_method]) eq 'RTMPT'
                                || uc($field[$pos_method]) eq 'HTTP'
                                || uc($field[$pos_method]) eq 'RTP' )

I then started reprocessing old logs like this:

/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=fms -LogFile=/oldlogs/access.30.log -showdropped -showcorrupted

And all is good in the world again.

Sep 062012
 

Haven’t posted in a while, but have been meaning to post this tidbit. We have been leasing a Prius V (a gas/electric hybrid) since April, and I decided to track our fuel economy. We got this car to replace our Yaris since it was a little cramped with a car seat and the massive stroller that comes along with becoming a parent.

If you are interested, you can see a spreadsheet of each time we purchased gas, the cost, kilometres travelled and average fuel economy over that period. The values in red are values that you can’t trust, as we were travelling, and I misplaced that receipt.

Here is the link: Google Spreadsheet

Mar 312012
 

A few weeks ago, I tossed Mint on an old machine to give it a try. Looking for something to replace my Ubuntu 10.04, and I’m not a fan of Unity at all. Mint, with its extensions to Gnome 3, seem to give the best of the new paradigm without changing everything I like about the old.

So last night I backup up my computer, and began the reinstall. I had issues even booting the Live DVD on my main computer. After some internet searching I resolved this by:

  • Pressing e at the bootup screen
  • Adding nomodeset to the configuration line before the —

I was then able to boot, and install the OS. This didn’t surprise me, as I’ve had to do similar tricks over the years to install OSes. Normally when everything is installed, we are good to go.

Oh boy, not this time. So after the install and reboot, I could not get X to load. Using the same trick as above, I was able to get into a GUI, but video performance was lack luster as no drivers were loaded. I attempted to install the restricted drivers, but they would install, and then on reboot, I’d have nothing again. I have noticed similar problems on other computers I have which have NVIDIA cards and AMD cpus. I was determined to find a fix this time. Instead of using the drivers from the repos, or the open source nouveau driver, I decided to go straight to the source, and download the driver from NVIDIA. You can’t install this from within X, and you will also need some extra packages, so before you start, make sure that:

  • You have all of the compilation tools installed: sudo apt-get install build-essential
  • And you have any header or development libraries that are needed for your kernel.

Once that is installed, you need to disable X. You cannot simply kill the X process, because it will start itself back up, and you can’t really run in runmode 3 or 1 as in the Debian world, 2 – 5 seem to be the same, and 1 may not run some of the system services required to install the driver. Many earlier tutorials mention running these commands to stop X for gnome:

  • sudo /etc/init.d/gdm stop
  • sudo /etc/init.d/gdm3 stop

Or KDE:

  • sudo /etc/init.d/kdm stop

But on mint, you need to use:

  • sudo /etc/init.d/lightdm stop

Once I ran that, I was dropped to a terminal, and I could run the NVIDIA installer as per the instructions. It disabled the nouveau driver and compiled my driver. After a reboot, all was good in the world.

I have a few thoughts on a simpler process. It is possible that the nouveau driver was conflicting with the NVIDIA driver. If I just disabled, or blacklisted the nouveau driver, the restricted drivers install wizard may have worked fine.

Here are a list of the pages I consulted to come to my resolution:

Feb 262012
 

One of my long time goals has been to be able to easily display tournament brackets on the web. It seems to be a difficult problem to solve with lots of questions out there, and few answers. There are a few solutions, but they have various issues and downfalls.

I decided to give it another try myself. And this is what I have come up with: A bracket builder, a saved file with raw data, and a bracket viewer that can read the files. If all that sounds like too much, the bracket builder can just spit out the HTML code to display the tree.

I’ve outlined how it works, uploaded an example and have provided a download so you can integrate it with your own applications.