Archive for the ‘General’ Category

Here are some neat database commands I recently came across while I was trying to get statistics and status information on some mysql and postgres databases.

Postgres

This will give you a list of all of the databases in the server.*
select datname from pg_database;

* If you don’t know how to connect to postgres from the command line, don’t fear (well fear a little). What I had to do was su to the user that is a DB user, and then run /usr/local/pgsql/bin/psql This launched a command line client (without needing a password), that let me query the server. I can’t provide a better understanding then that, as this is the first time I’ve ever interacted with a postgres server.

This tells you how big a particular database is.
SELECT pg_size_pretty(pg_database_size('xythosdocs'));
The output is like this:
pg_size_pretty
----------------
2834 MB
(1 row)


Mysql

Get all database sizes:
SELECT table_schema "Data Base Name", sum( data_length + index_length ) / 1024 / 1024 "Data Base Size in MB", sum( data_free )/ 1024 / 1024 "Free Space in MB" FROM information_schema.TABLES GROUP BY table_schema;
Produces output like this on the command line:

+--------------------+----------------------+------------------+
| Data Base Name     | Data Base Size in MB | Free Space in MB |
+--------------------+----------------------+------------------+
| blogadmin          |           0.35750008 |       0.00098896 |
| blogs              |          90.60342026 |      27.73141003 |
| information_schema |           0.00781250 |       0.00000000 |
| musicschedule      |           0.00548172 |       0.00020599 |
| mysql              |           0.54032421 |       0.00000000 |
| phpflash           |           0.77745819 |       0.00000000 |
+--------------------+----------------------+------------------+

Get some status information on the sever. Run this from the command line:
mysqladmin -uroot -p status
Phpmyadmin also provides a lot of process status information if you have that installed.

I’m in the process of migrating multiple blogs into a single installation of WordPressMU. Please stand by as I correct the problems here.

In the end, I promise to have a nice blog post on it.

I recently installed the excellent OpenShot video editor for Ubuntu because I had to splice together some video from our curling league. We had two cameras shooting the action from the Semi-Final, and Final games.

OpenShot is a very easy to use program from making some basic movies and it handled the two different format videos pretty well.

However, after installing, I noticed that I could no longer use AMSN, my instant messenger of choice. When launching from the terminal, I would receive an error about GStreamer problems. This also affected Pidgin, and I’ve heard reports of problems with Empathy and others too. The problem lies in frei0r-plugins package, which messes up gstreamer for some programs, and is required by OpenShot. The easy solution to get it working is to remove frei0r-plugins, but of course, OpenShot won’t work anymore.
sudo apt-get purge frei0r-plugins

After searching the net a bit, I ran across a launchpad page that pointed the finger at libgavl. The problem is fixed in libgavl 1.1.1. Of course, 1.1.0 is the highest version you can get with Ubuntu 9.10.

So, to correct this once and for all, you can compile 1.1.1 yourself, or you can download the .deb for Ubuntu 10.04. So, go here: http://packages.ubuntu.com/lucid/libgavl1 and download the correct version for your architecture, and you should be good to do.

I got some warnings about an older, more supported version being available in the repositories, but you can safely ignore that.

Just surfing the web and came across this excellent site with advanced awstats configuration information. As you all know, I do enjoy using awstats, and have had to modify it in the past.

Well, this site has lots of cool scripts and modifications to make awstats more useful: http://www.internetofficer.com/awstats/

My favourite modifications so far are:

• Bing in awstats
• Detecting Hotlinking

And don’t forget to check out my own tutorial on adding robots (Blog Post/Wiki Entry)

I’ve configured awstats to provide statistics on visitors of the websites on the server. I’ve also configured xymon to notify me when the website is not accessible. xymon uses a bot to check every 5 minutes to make sure the site is still up. As you can imagine, this generates a lot of hits, that aren’t really people.

So, since I don’t want to disable monitoring, I thought I should at least configure my stats so that they reflect people, and this bot.

awstats has an excellent robot detecting feature, but they do not yet detect xymon. So, in order to make the change, you need to dive into the robots.pm file, normally saved in /usr/local/awstats/wwwroot/cgi-bin/lib/

You will need to make two changes for it to detect this new robot (it tells you this at the top of the file). So, at the end of the @RobotsSearchIDOrder_list1 = ( array, I added:

• ‘Hobbit[_+ ]bbtest-net/4.2.3′

And then at the top of Other robots reported by users in %RobotsHashIDLib = (, I added:

• ‘Hobbit[_+ ]bbtest-net/4.2.3′,’xymon’,

Once I deleted the files in /var/lib/awstats/ and reprocessed my logs, the statistics better reflected my visitors. Beware that if you delete files from /var/lib/awstats/ and don’t have the original log files around, you will lose the history for that period of time. Also, this change should be reflected on all sites that you have running on this server and using awstats.

My webserver only had logs for August, so I only deleted the file for August, and left the file for July with the missed detections since I cannot rebuild it. I’ve since changed the configuration of the log rotater.

Later I’ll be posting a more in depth tutorial on installing awstats, and also on modifying the robots.pm file, since I couldn’t find much about it online.

UPDATE: As promised, here is a more detailed how-to on updating the robots file: http://wiki.cornempire.net/doku.php?id=awstats:awstatsrobots

I just noticed this today. I monitor a number of sites and servers using the great Xymon system monitoring tool (Linky). It is based on early Big Brother code. Anyway, today I noticed that the graphs were all blue. Something like this:

Bad Xymon Graph

I’m using Xymon 4.2.3 on the servers, and both servers show the graph like this using Firefox 3.5 on Linux. Google Chrome development version, SeaMonkey 1.1.5, Epiphany and Galeon all display the graph correctly.

I have no solution, but I thought I’d post my experience.

Note:

Instead of editing my previous post to pieces, I thought it might be best if I repost my script with a better description of how it works. I’ve written up documentation of the script on the CornEmpire Software wiki which is available at http://wiki.cornempire.net. The documentation directly related to this script is available at http://wiki.cornempire.net/doku.php?id=d2l:bulkdeactivate

Introduction

One of the often requested features is to be able to bulk deactivate courses in D2L after they have completed. You can currently deactivate courses, but this is a one at a time effort. This has become more important as version 8.3 of D2L brought along a new My Courses widget, which allows users to see updates from each of their courses. Unfortunately, when course access ends, they can still see the updates. This has caused confusion for many users.
Another reason to deactivate courses is minimize the amount of clutter in the view of users. If course access has ended, and they cannot access the course, there is no need for the course to remain active (this could vary depending on how your roles are configured). A way around this is to script the bulk deactivation of courses. Through the use of some Javascript coding, we can instruct the web browser to visit each course, and deactivate it for us.

See the script in action.

Would you like to see more? I have the script, and it’s documentation available on this page: http://wiki.cornempire.net/doku.php?id=d2l:bulkdeactivate

If you have any questions, please post them below.

Update

Please see my new post on this topic. Available at: http://www.cornempire.net/2009/07/07/d2l-bulk-deactivation-part-2/

Introduction

One of the often requested features is to be able to bulk deactivate courses in D2L after they have completed. You can currently deactivate courses, but this is a one at a time effort. This has become more important as version 8.3 of D2L brought along a new My Courses widget, which allows users to see updates from each of their courses. Unfortunately, when course access ends, they can still see the updates. This has caused confusion for many users.

Another reason to deactivate courses is minimize the amount of clutter in the view of users. If course access has ended, and they cannot access the course, there is no need for the course to remain active (this could vary depending on how your roles are configured). A way around this is to script the bulk deactivation of courses. Through the use of some Javascript coding, we can instruct the web browser to visit each course, and deactivate it for us.

Example

Take a look here for this script in action.

If you think that this might be useful for your organization, you can download the html file which includes the javascript.

For D2L Version 8.3
For D2L Version 8.4 (Thanks Dennis!)

Instructions on Use

• You will need to edit the file and modify line 5. This contains the URL to your learning environment, and the URL to your Course Offering Information screen.
• This file will need to be placed in one of your courses in your learning environment in the Manage Files area.
• You need to open the file from your Manage Files section and provide the script with the OU numbers of the courses you need to deactivate. (Take a read below to see how we get the ou numbers)
• Once the OU numbers are loaded, you can click the Start Processing button and your web browser should go to all the courses and deactivate them.

Limitations

• You are required to have administrative access to any courses you need to deactivate (I haven’t been able to figure out why this is, but a javascript error is thrown if you do not have admin access)
• This has been tested in Firefox 3 on Windows XP and Ubuntu 9.04 as well as a Chrome Development Beta for Linux but may not work in other web browsers (especially IE).
• This has been tested on D2L Version 8.3 MR01. I’d suspect it would work on any version of 8.3 but not sure about 8.4 yet, or any earlier versions.
• I’m not responsible if this code melts your CPU. or causes any other horrible damage to your D2L installation (although it shouldn’t….it’s only about 35 lines of Javascript. :p)

Getting a List Of Courses

The tricky part in all of this is actually getting the OU numbers for each of the courses you need to deactivate. For those who don’t know, the OU number is the org unit number, and is an internal identifier for D2L. You will notice the ou number in the URL of courses as ou=#####.

We run a report using the reporting tool that gives us the ou numbers (in one, non-ideal form. Hopefully D2L web services will eliminate the need for this).

In the Reporting tool, create a report using the Org Units dataset. With in the report, include the column for the course Path. This path contains the OU number for the course. Under the filter tab, add a few filters that will narrow down your search to the information you require. Our organization adds a semester code to all of our courses, so all courses that end in 200802 are courses that ran in the last semester. We can query these courses, and get a list of the ou numbers through the course offering paths. See below for some images of our report in action.

Then you have to do some work in manipulating the report to filter out the OU numbers. I select the whole column using firefox, and dump it into OpenOffice. Customize the import to break the lines on all of the dividing characters, and then I have a column of ou numbers. Then take that, and add in commas after all of the entries using a find and replace. You will need a text editor that can find and replace on characters you cannot see. Replace the new lines with commas.

Hopefully you can find a better way to get your list of OU numbers . If you have a good way, leave a comment and let me know.

I recently updated my server and I store a good number of passwords in my pwman database. It is a great little command line program for storing passwords in a secure manner. I do this so that I can log into my machine from anywhere in the world to retrieve passwords if needed, and I’m not tied to a thumb drive that I could lose/break, or a computer that may not have the necessary OS/ports for me to use.

If you need to migrate your database, you can follow these steps:

1. Grab a copy of your old database. It is usually called pwman.db and is stored in your /home/yourname/.pwman/ If you can’t find it, there is a config file in this directory that will tell you where it is.
2. Install pwman3 on your new server. (command for Debian based Linux OSes)
   apt-get install pwman3
3. Run pwman3 and configure the same master password as your old database. (Not sure if this is required, but I thought it was a good idea)
4. Make a copy of your database, and then put your old one in your new /home/yourname/.pwman/ folder.

The next time you run pwman3, you will have all your old passwords!

In my last few attempts at configuring my mythtv installation (my most recent being today when I replaced the hard drive), I’ve gotten this error when I log into my box through SSH:

MythTV status for localhost
===========================
Status...........: Wed Feb 18 2009, 8:30 PM
Total Disk Space.: Total space is 548,770 MB, with 158,049 MB used (28.8%)
Next Recording In: 2 Hours, 59 Minutes
 
Encoders:
mythbox (1) - Idle
 
Scheduled Recordings:
2009-02-18 23:30:00 - CSI: NY (WBZ)
2009-02-19 00:30:00 - Stargate SG-1 (Space: The Imagination Station)
2009-02-19 10:30:00 - Stargate SG-1 (Space: The Imagination Station)
2009-02-19 12:30:00 - First Wave (Space: The Imagination Station)
2009-02-19 19:30:00 - Coronation Street (CBNT CBC St. John's)
2009-02-19 22:30:00 - CSI: Crime Scene Investigation (WBZ)
 
Schedule Conflicts:
Unable to access MythTV Perl API. Try with --verbose to find out why.

I get most of the status for the server, but I don’t get the read outs of the conflicts because of the error.

What happens (as far as I understand it) is that when the cronjob generates the motd, it runs as root, and root does not have a link to the mythtv config, so it can’t get the data from the database.  So, to get around this, we need to create a link in /root/.mythtv/ to the config.xml file for the server.  For me, this runs from my home directory /home/thomas/.mythtv/config.xml so:

sudo ln -s /home/thomas/.mythtv/config.xml /root/config.xml


Once you make this change, the next time the cronjob runs it should populate the motd correctly.

Notes: I’m running this server on Ubuntu 8.04.02 64-bit.

You can find more information on this issue here.