Why would one want to do this, you may ask? Well, for starters, it makes a great ‘canary-in-the-coal-mine‘ in regards to backwards compatibility!
For Drizzle, we’ve created some tables (via the randgen’s data generator if you are curious), saved a copy of the datadir, and then created a test case that uses said datadir for the test server. The test executes some simple SQL queries to make sure we can read the tables properly. This way, if we ever do something to either the server or .dfe format (data format exchange – had a most enlightening conversation with the team about this format’s history at the MySQL UC), we’ll have a broken test that cries about it. From there, we’ll know we have to take some action. The always-amazing Stewart Smith has also created some foreign key backwards compatibility tests, which I believe marks further progress towards the magical goodness that is catalogs!
We signal that we want to do this by using a .cnf file:
[test_servers]
servers = [[]]
[s0]
load-datadir=backwards_compat_data
Each server is named s0,s1,sN. If a server name is contained in the .cnf file, the test-runner will do the appropriate magic to use the specified datadir for that server. The argument to load-datadir is the name of the directory that is intended for use in the test. All datadirs are expected to live in drizzle/tests/std_data. Tests that do use a .cnf file, like main.backwards_compatibility and slave.basic are skipped by test-run.pl automatically (you *can’t* run them via test-run.pl).
This is something that I don’t believe could be accomplished with the old test runner, or at least not *easily* done (see Rube Goldberg) ; ). At some point, we will switch over to dbqp entirely and remove test-run.pl. Seeing comments like this, makes me happy and think things are on track.
dbqp was created with the idea that it should be easy to express complex testing setups (multiple servers, using a preloaded datadir, etc, etc) and it looks like the incubation is starting to pay some benefits. In addition to allowing this voodoo to happen, the code I’ve added to the test runner will allow us to start doing proper tests of the super Mr. Shrewsbury’s multi-master replication. Joe Daly has also been doing some very promising work for hierarchical replication based on Dave’s tree. I’ll be creating some example tests for these badass features soon. The moral of the story is that by rethinking our test-runner, one tiny bit of code helps us move the ball forward on testing replication, backwards compatibility, and catalogs.
It’s honestly one of the best parts of working on the Drizzle project – being encouraged to experiment and rethink problems has enabled all sorts of innovation (but one example of Monty Taylor’s computing wizardry!) and cool features. Thanks to this freedom to experiment, we now have even more ways of making sure we are producing quality code.
My view of QA is that we do help test, but that we also help other people answer their own questions about quality (via tools, documentation, examples, etc). Ultimately, a test is a question – “Do you return the right answer for this query?”, “Can you survive a beating from the randgen?”, etc – and asking questions should be easy and informative. QA shouldn’t be the sole province of some obscure priest class, but everyone’s playground. When I see developers like Stewart writing interesting test cases and even contributing to the test tool itself, I’m even happier than when I find a bug (and finding bugs is quite awesome!).
Anyway, the code is proposed for a merge to trunk and documentation is available (under testing/writing test cases). I hope that this makes trying to break things even more fun for people >: )
