Blog | Bugs | Build Farm | Events | Feature Requests | IRC | LCOV | Mailing List | Planet Drizzle | Source Code | Source Docs | Wiki
This page serves to document how to create a new test case for Drizzle.
Contents |
Test Cases for Plugins
Compiling a Plugin into the Server
Currently, some plugins (non-storage engine, non-thread scheduler) plugins require a different compile than normal. To compile the server with support for one of these plugins, do:
$> ./configure --with-$pluginname-plugin $> make
Setting up a new Test Suite for your Plugin
When developing plugins, it is important to create test cases which verify the SQL interface for your plugin behaves as documented and expected. The first step in creating a test case for your plugin is to create a suite in the tests/ directory:
$> cd tests $> mkdir -p suites/$pluginname $> cd suites/$pluginname $> mkdir -p t $> mkdir -p r
What this does is create all the required directories in your plugin's test suite. The t directory will contain your test case files and the r directory will contain the result files.
Next, you will want to add this new directory to the bzr tree:
$> bzr add $pluginname
Creating the master server's startup file
When the test-run.pl script runs a test case, it first looks for a master.opt file with the same name as the test case it needs to run. It uses this master.opt file to provide startup parameters to the server used during the processing of the test case.
Because Drizzle loads and registers plugins upon startup, test cases for plugins usually will be accompanied by a $testname-master.opt file. I say usually because certain plugins such as storage engines currently are automatically loaded on server startup.
This file contains, on a single line, the startup parameters that the master server uses when running the plugin's test cases. To test a plugin, clearly the plugin must be enabled on server startup. Enabling a plugin is as simple as supplying the server with an --plugin_load=$plugin_lib parameter.
Let's do this now for the MD5 UDF plugin. Change to the t directory and create the master.opt file:
$> cd t $> touch md5-master.opt $> vim md5-master.opt
Now, in the opt file, put this:
--plugin_load=libmd5udf.so
How do I know the name of the shared object library used by the plugin? Easy: take a look at the Makefile.am in the plugin's source directory:
$> cd $basedir/plugin/md5 $> cat Makefile.am | grep lib
On my machine, this produces:
jpipes@serialcoder:~/repos/drizzle/working/plugin/md5$ cat Makefile.am | grep lib
EXTRA_LTLIBRARIES= libmd5udf.la
libmd5udf_la_LDFLAGS= -module -avoid-version -rpath $(pkgplugindir)
libmd5udf_la_LIBADD= $(CRYPTO_LIBS)
libmd5udf_la_CPPFLAGS= ${AM_CPPFLAGS} -DDRIZZLE_DYNAMIC_PLUGIN
libmd5udf_la_SOURCES= md5udf.cc
EXTRA_LIBRARIES= libmd5udf.a
libmd5udf_a_SOURCES= md5udf.cc
In that output, you see that the name of the shared object library will be called libmd5udf[.a|.la]. When libtool "glues together" the produced object file when this plugin is compiled, a library called libmd5udf.so will be put in the .libs directory in the plugin source directory. The --plugin_load=$libname directive on server startup directs the server to look for $libname in a plugin source directory and dynamically load the object file into the server's address space.
OK, now that the master.opt file is created, let's now work on a simple test case.
Creating Your First Test Case
Test cases for plugins should follow these basic principles:
- All documented behaviour of a plugin should be examined
- Comments should be applied liberally to ensure that future developers understand precisely what is being tested. If a specific test piece verifies a bug fix, a link to the bug on Launchpad should always be provided in the comments above the test
- All results should be deterministic -- this means that every run of the test should result in identical results. In cases where non-deterministic results are expected (not common) there are a few test suite commands to deal with these situations (detailed below)
- Group SQL code and test commands into segments which correspond to the functionality which the group validates
OK, let's go ahead and walk through a sample test case we'll create for the MD5 UDF plugin.
The purpose of the MD5 plugin is to create a 32-byte deterministic hash from a series of raw bytes. Therefore, a test case for the plugin must verify:
- That successive calls to the MD5 function when supplied with the same sequence of characters always produces identical results.
- That the result of calls to the MD5 function always produce a 32-byte value
- That the format of the MD5 function's output corresponds to a hexidecimal common notation
- That the output of the MD5 function is identical when supplied with a constant or a field value containing the same constant
- That the output of the MD5 function when supplied with various values is consistent but distributed
Create a test case file in the t directory and open it in an editor:
$> touch md5.test $> vim md5.test
At the top of the test case file, we put this:
#
# Verify that MD5() function returns a deterministic result
# when successively supplied with the same constant.
#
SELECT MD5("I love testing");
SELECT MD5("I love testing");
Good enough for now. Let's run our new test case:
$> ./dtr --suite=md5 md5
On my machine, this is the output we get:
jpipes@serialcoder:~/repos/drizzle/working/tests$ ./dtr --suite=md5 md5
Logging: ./dtr --suite=md5 md5
<snip>
=======================================================
DEFAULT STORAGE ENGINE: innodb
TEST RESULT TIME (ms)
-------------------------------------------------------
md5.md5 [ fail ]
drizzletest: The specified result file does not exist: '/home/jpipes/repos/drizzle/working/tests/suite/md5/r/md5.result'
The result from queries just before the failure was:
SELECT MD5("I love testing");
MD5("I love testing")
dc30ccbf874c4faa408ffdc500c2e29a
SELECT MD5("I love testing");
MD5("I love testing")
dc30ccbf874c4faa408ffdc500c2e29a
Excellent. Our test run failed because we have not yet created a result file, but we can see that the repeated calls to MD5("I love testing") produced identical results.
Recording the Results File
Let's go ahead and record a results file now before going further. To do so, we run the same command, but with the --record option:
$> ./dtr --suite=md5 --record md5
On my machine, we get:
jpipes@serialcoder:~/repos/drizzle/working/tests$ ./dtr --suite=md5 --record md5 Logging: ./dtr --suite=md5 --record md5 <snip> ======================================================= DEFAULT STORAGE ENGINE: innodb TEST RESULT TIME (ms) ------------------------------------------------------- md5.md5 [ pass ] 25 ------------------------------------------------------- Stopping All Servers All 1 tests were successful. The servers were restarted 1 times Spent 0.025 of 4 seconds executing testcases
This will "record" your test case results into a file named /tests/suite/$pluginname/r/$testname.result. The results is both the SQL statements executed as well as the results from those SQL statements, as you can see in the output below:
jpipes@serialcoder:~/repos/drizzle/working/tests$ cat suite/md5/r/md5.result
SELECT MD5("I love testing");
MD5("I love testing")
dc30ccbf874c4faa408ffdc500c2e29a
SELECT MD5("I love testing");
MD5("I love testing")
dc30ccbf874c4faa408ffdc500c2e29a
Add Your Test Case and Result File to Bazaar
Before going any further, go ahead and add your test case, master.opt file and result file to Bazaar:
$> bzr add suite/md5/t/md5.test suite/md5/t/md5-master.opt suite/md5/r/md5.result
Expanding Your Test Case
At this point, you will start adding additional tests to your test case which stress various things that your plugin does. We'll continue here with our MD5 UDF plugin example and enhance the test case with a few more tests.
Let's add a simple test to verify that the MD5 of a constant is identical to the same constant stored in a table in a database.
Add the following to the bottom of /tests/suite/md5/t/md5.test:
#
# Check that the hash of a constant is identical to the hash
# of the same constant stored in a table column.
#
--disable_warnings
DROP TABLE IF EXISTS t1;
--enable_warnings
CREATE TABLE t1
(
some_text VARCHAR(100) NOT NULL
);
INSERT INTO t1 VALUES ("I love testing");
SELECT MD5("I love testing") = MD5(some_text) FROM t1;
DROP TABLE t1;
Here, we need to create a table in which to put the constant "I love testing" and then compare the hashed MD5 value of the stored value against a hash of a supplied constant.
We use the --disable_warnings and --enable_warnings drizzletest commands to indicate we don't care about warnings emitted if table t1 does not exist.
We then issue a CREATE TABLE statement with just enough to create the scenario we want to test, populate the table with a single row with the column some_text containing the value "I love testing".
Next, we issue a SELECT which compares the results of a MD5() of the constant against the value of MD5() of the stored value of the same constant. If a 1 is returned from the server for this comparison, that means the values are the same.
Let's see the output of our changes by re-running the test:
jpipes@serialcoder:~/repos/drizzle/working/tests$ ./dtr --suite=md5 md5
Logging: ./dtr --suite=md5 md5
<snip>
=======================================================
DEFAULT STORAGE ENGINE: innodb
TEST RESULT TIME (ms)
-------------------------------------------------------
md5.md5 [ fail ]
--- /home/jpipes/repos/drizzle/working/tests/suite/md5/r/md5.result 2009-05-14 23:31:54.000000000 +0300
+++ /home/jpipes/repos/drizzle/working/tests/suite/md5/r/md5.reject 2009-05-14 23:50:11.000000000 +0300
@@ -4,3 +4,13 @@
SELECT MD5("I love testing");
MD5("I love testing")
dc30ccbf874c4faa408ffdc500c2e29a
+DROP TABLE IF EXISTS t1;
+CREATE TABLE t1
+(
+some_text VARCHAR(100) NOT NULL
+);
+INSERT INTO t1 VALUES ("I love testing");
+SELECT MD5("I love testing") = MD5(some_text) FROM t1;
+MD5("I love testing") = MD5(some_text)
+1
+DROP TABLE t1;
drizzletest: Result length mismatch
Aborting: md5.md5 failed in default mode.
To continue, re-run with '--force'.
Stopping All Servers
As you can see, a "1" was produced as the result of our SELECT comparison. This means that the expected behaviour of identical hashing of a stored string and a supplied constant string is produced. This is a validation of a correct test result, and so we re-record the test to store our results:
$> ./dtr --suite=md5 --record md5
In this way, we make small, validated changes to our test case in an iterative approach, allowing test cases to be enhance and expanded in a controlled manner.
Also note that each test "segment" is accompanied by a comment explaining the purpose of the test commands.