A Login Debugging Tool for PlanetLab Another "Co" tool from the CoDeeN project

What
Is It?

CoTest is a login debugging tool for PlanetLab. If you are having problems logging into a node, you can run CoTest to see what various data sources think about the node in question. The output is meant to be human-friendly. This tool gets its inspiration from Neil Spring's "why" script.

How Does
It Work?

You download CoTest, compile it, and run it on the command line. You provide it your slice name, and a list of nodes, and for each node, it provides some information about any problems the node is experiencing. It currently pulls data from two sources - the CoMon and CoTop. It uses a fairly simple process to determine what problems might be occurring, and while it's not perfect, it should be reasonably accurate.

Installing
CoTest

Follow these steps to install CoTest on your own system:

1. Download cotest-0.8a.tar.gz, which contains the necessary files
2. Expand the file by running
   gunzip cotest-0.8a.tar.gz
   followed by
   tar xvf cotest-0.8a.tar
   You should now have a directory called cotest-0.8a
3. Inside that directory, you will find two files - cotest.c and Makefile. If you are on a Sun system, edit the Makefile and uncomment the line specified. If you are running Linux or FreeBSD, you should not need any changes. Now, you are ready to build CoTest. Simply type
   make
   and you should have an executable called cotest. You can put this file somewhere in your path if you would like to make it easier to invoke, or you can leave it where it is.
4. You are now ready to use CoTest.

Running
CoTest

CoTest is a fairly simple program to use. Simply type
./cotest slice_name node.name
where slice_name is the name of your slice, and node.name is the fully-qualified name of the PlanetLab node that you are trying to test. For example, if our slice is princeton_codeen, and our node is planetlab-1.cs.princeton.edu, we would type

./cotest princeton_codeen planetlab-1.cs.princeton.edu

We may see results like the following:

cotest version 0.8a - see http://codeen.cs.princeton.edu/cotest for updates
Status: current version 0.8a released Apr 28,2006
Using server: summer.cs.princeton.edu
planetlab-1.cs.princeton.edu: both CoMon and CoTop think node is dead

In this case, all of the tools think the node is dead, so the node is likely to be dead at the moment. We can also get different information if we choose other nodes. If we run

./cotest princeton_codeen planetlab1.it.uts.edu.au

we may see something that looks like the following:

cotest version 0.8a - see http://codeen.cs.princeton.edu/cotest for updates
Status: current version 0.8a released Apr 28,2006
Using server: summer.cs.princeton.edu
planetlab1.it.uts.edu.au: just fyi - clock drift 18510 secs
planetlab1.it.uts.edu.au: slice princeton_codeen exists

In this case, CoTest is telling us that other than the node's clock being out of sync, the node looks normal.

CoTest tries to simplify a number of conditions monitored by the tools that it uses. The list of messages that it can produce include the following, but some of these messages may not be in use at the moment because the PSEPR tools is not being polled:

• administrative ssh failing - This is generally a bad sign, and may indicate problems with the node.
• both CoMon and CoTop think node is dead - Neither CoMon nor CoTop can access the node - if PEPR also sees the same problem, the node is probably dead or offline.
• CoMon failing, but CoTop ok - This may just be an error with CoMon, but it might indicate the node is acting strangely.
• CoMon seems to think the node's OK - As far as CoMon can tell, the node is fine, so the problem may be related to your slice.
• could not find created slice - Your slice does not appear to have been created on this node. Check to make sure that you have in fact assigned this node to your slice.
• ERROR - couldn't find list of slices - This is either a problem with CoTest or PEPR. Make sure you are running the latest version of CoTest.
• extra slice - Your slice is listed as being extra, meaning that the node believes it should not exist.
• failed getting node info - PEPR does not have info on this node.
• heavily loaded - load XX -The node may be running slow due to its load, but it otherwise appears functional.
• heavily loaded, probably frozen - The node appears to have stopped responding due to overload. Chances are good that the node needs to be rebooted.
• just fyi - clock drift XX secs - The node's clock seems to be out of sync. Generally not a fatal condition.
• last ssh X hours ago, very suspicious - The CoMon slice is generally tested for ssh at least once an hour, and this machine doesn't seem to have been used in that time. This may indicate it's not possible to log into the machine
• missing slice - Your slice is listed as being missing on the node, which tends to indicate a problem with the node's management software.
• name lookup failed - The node does not seem to be resolvable via DNS. Chances are you typed the wrong name.
• no results in CoMon - CoMon does not have any data for this node. It may simply be that CoMon is not aware of this node yet.
• node has @ sign - skipping - Probably a typo.
• node http failing - not fatal, but suspicious - Each node runs a web server, which should run all of the time. If it is not running, there may be other problems with the node.
• node name missing dots - skipping - All node names need to be fully-qualified, which means all dots need to be included.
• PEPR thinks node down - The PEPR database thinks the node is down. If this is confirmed by CoMon, assume the node is dead.
• probably frozen - CoMon running, CoTop failing, drift XX secs - In some failure cases, the node is basically frozen, but the CoMon program continues to run. The node may be unusable.
• slice %s exists - Your slice exists and appears normal. You should be able to log in.
• slightly suspicious - CoMon running, CoTop failing - This may just indicate some problem with CoTop.
• ssh failing - If the node's ssh is failing, then you will not be able to log into it.
• your clock may be off - Comparing your machine's local time to the last login time for the CoMon slice indicates that your machine is backward in time, something that should not happen unless your own clock is skewed.

Tips &
Shotcuts

CoTest is designed to work in conjunction with our other tools. If you specify your slice name via the MQ_SLICE environment variable, you will not need to specify it on the command line.

You may want to run CoTest on multiple nodes. You have two options - you can specify all of them on the command line, or you can specify none of them on the command line. If no nodes are listed on the command line, then it is assumed you will provide them on standard input. If you are typing them manually, the program can be terminated by typing ctrl-D.

Finally, CoTest keeps some cached information in the directory cotest_cache in your home directory. PEPR alert information is cached for one hour, and CoMon node information is cached for five minutes. If you believe these to be stale, you may manually delete this directory.

Status
& Bugs

CoTest is currently in beta testing. Please let us know if you encounter anything that seems strange or confusing. Likewise, feel free to suggest any improvements.

People

Vivek Pai, with help from KyoungSoo Park and input from lots of others. We may collectively be contacted at princeton_codeen at slices.planet-lab.org

We would like to thank Neil Spring for his original "why" script, which motivated this effort. We would also like to thank Mic Bowman and the folks at Intel Oregon for PEPR, which we use as a data source.