Wednesday, December 15, 2010

How to run JUnit tests with a timeout

For my main project, OpenSHA, we have operational JUnit tests that test system status on an hourly basis (through cruise control). For these tests, we want to make sure that all core services are operational and responsive. To do this, I needed a way to run JUnit tests with a timeout.

For example, for a map making request, it should fail if it takes over 120 seconds (which is more than generous).

I didn't find a built in way to do this, so I created my own with threads and java reflection. To use it, you call a method like this:


@Test
public void doTest() throws Throwable {
TestUtils.runTestWithTimer("sleepTestMethod", this, 2);
}


where "sleepTestMethod" is the method that gets run as a JUnit test. Note that "sleepTestMethod" shouldn't have the @Test annotation, because then it will be run twice.

Here is an example failure:

junit.framework.AssertionFailedError: method 'runTest' exceeded timeout of 120 secs!
at util.TestUtils.runTestWithTimer(TestUtils.java:70)
at org.opensha.commons.mapping.gmt.TestGMT_Operational.testMakeMapUsingServletGMT_MapStringString(TestGMT_Operational.java:37)


Here is the code for TestUtils:


package util;

import static org.junit.Assert.fail;

import java.lang.reflect.InvocationTargetException;
import java.lang.reflect.Method;

public class TestUtils {

private static class TestMethodThread implements Runnable {

private Method testMethod;
private Object testObj;

private Throwable exception;

public TestMethodThread(Method testMethod, Object testObj) {
this.testMethod = testMethod;
this.testObj = testObj;
}

@Override
public void run() {
try {
testMethod.invoke(testObj);
} catch (Throwable t) {
this.exception = t;
}
}

public Throwable getException() {
return exception;
}

}

/**
* This runs a JUnit 4 test method in a separate thread with a timer. If the timeout is
* exceeded, then the test fails.
*
* @param methodName the name of the test method
* @param testObj the object for which method methodName is to be called
* @param timeoutSeconds timeout in seconds until the test should fail
* @throws Throwable
*/
public static void runTestWithTimer(String methodName, Object testObj, int timeoutSeconds) throws Throwable {
// get the method
Method testMethod = testObj.getClass().getDeclaredMethod(methodName);

// make sure it's accessible (not private)
if (!testMethod.isAccessible())
testMethod.setAccessible(true);

// create the thread which will simply run this test
TestMethodThread testThread = new TestMethodThread(testMethod, testObj);
// start the thread
Thread t = new Thread(testThread);
t.start();
// record the start time in milis
long start = System.currentTimeMillis();

while (t.isAlive()) {
// seconds that the thread has been running
double timeSecs = (double)(System.currentTimeMillis() - start) / 1000d;
if (timeSecs > timeoutSeconds) {
// if we're here, then it's exceeded it's allotted time.
try {
// try calling interrupt to end any blocking operation
t.interrupt();
} catch (Throwable e) {
e.printStackTrace();
}
// now fail
fail("method '"+methodName+"' exceeded timeout of "+timeoutSeconds+" secs!");
}

// if we're here then it's still running, but within the time limit. Sleep for 500 milis
try {
Thread.sleep(500);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
// see if it ended with an exception, this exception might be an assertion failed exception
Throwable exception = testThread.getException();
if (exception != null) {
Throwable cause = exception.getCause();
// if it's an assertion error, throw that so the failure shows up nicely in JUnit as opposed
// to an error.
if (cause != null &&
(cause instanceof AssertionError || exception instanceof InvocationTargetException))
throw cause;
// otherwise it actually is an error (not a failure), throw the exception
throw exception;
}
}

}



I hope this is useful to someone!

1 comment:

  1. With JUnit4 you can use
    @Test(timeout = 120000)
    for a built-in timeout check.

    ReplyDelete