Category Archives: Drizzle

Posted on by

dbqp / randgen docs are up

As I mentioned earlier, dbqp now has a randgen mode.  This allows us to run test cases for the randgen in much the same way as we execute drizzletest cases.  At present, all of our randgen tests are defined as dbqp cases and live in drizzle/tests/randgen_tests.  They are organized into suites (much like drizzletest cases).  You can also use it as a quick means of creating an ad hoc replication setup!

 

Anyway, we now have proper documentation up.  Instructions and prereqs for the randgen are included in the docs.  If anyone runs into any issues, please write a bug and I’ll get on it.  Thanks and have fun testing! : )

 

We have lots more cool stuff planned for dbqp and drizzle.  Ultimately, the goal is to make it fast, easy and efficient for anyone to make sure the server is behaving as they expect it to – we want happy users!  With that said –   GA release tomorrow!  It is a huge deal for us and lots of blood, sweat and tears have gone into the software.  We are really excited and hope that everyone else will wonder how they lived without Drizzle for so long (I know I have) : )

 

Posted on by

dbqp / randgen integration…huzzah!

What is the big deal, you may ask?  Well, read on and all shall be revealed, intrepid reader ; )
As I mentioned an earlier post, our new test-runner – dbqp – allows us to define testing ‘modes’ which all utilize the same system and server management code.

One only has to define a testManager (what does a test look like / how to organize tests) and a testExecutor (how to execute / evaluate a test).  The aim is for dbqp to be a one-stop shop for test execution and to provide a clean and simple way to manage and expand this.

I have just added –mode=randgen to the test-runner.  The random query generator is a significant part of Drizzle’s testing strategy and we use a large number of tests with the tool.  Currently, they are executed and managed by drizzle-automation / Jenkins in our build system, but there has not been an easy way for users to execute these tests (outside of installing / learning drizzle automation or the randgen)…until now >; )

Documentation including requirements and setup instructions for the randgen can be found here
For the cost of about 300 lines of code (the new manager/executor files), we can now execute our randgen tests much the same way as we do our drizzletest suite!

./dbqp –mode=randgen –randgen-path=/home/user/repos/randgen
<snip>
25 Feb 2011 12:14:55 INFO: Using testing mode: randgen
25 Feb 2011 12:14:55 INFO: Processing test suites…
25 Feb 2011 12:14:55 INFO: Found 18 test(s) for execution
25 Feb 2011 12:14:55 INFO: Creating 1 testbot(s)
25 Feb 2011 12:14:55 INFO: Taking clean db snapshot…
25 Feb 2011 12:14:55 INFO: testbot0 server:
25 Feb 2011 12:14:55 INFO: NAME: server0
25 Feb 2011 12:14:55 INFO: MASTER_PORT: 9306
25 Feb 2011 12:14:55 INFO: DRIZZLE_TCP_PORT: 9307
25 Feb 2011 12:14:55 INFO: MC_PORT: 9308
25 Feb 2011 12:14:55 INFO: PBMS_PORT: 9309
25 Feb 2011 12:14:55 INFO: RABBITMQ_NODE_PORT: 9310
25 Feb 2011 12:14:55 INFO: VARDIR: drizzle/tests/workdir/testbot0/server0/var
25 Feb 2011 12:14:55 INFO: STATUS: 1
25 Feb 2011 12:15:00 : ================================================================================
25 Feb 2011 12:15:00 : TEST NAME                                               [ RESULT ]    TIME (ms)
25 Feb 2011 12:15:00 : ================================================================================
25 Feb 2011 12:15:00 : main.basic                                              [ pass ]         4376
25 Feb 2011 12:15:08 : main.blob                                               [ pass ]         7829
25 Feb 2011 12:15:12 : main.collations                                         [ pass ]         3989
25 Feb 2011 12:15:16 : main.combinations                                       [ pass ]         4169
25 Feb 2011 12:15:19 : main.create_drop                                        [ pass ]         3387
25 Feb 2011 12:15:34 : main.drizzledump_restore                                [ pass ]        14726
25 Feb 2011 12:15:42 : main.drizzledump_restore_rand                           [ pass ]         8289
25 Feb 2011 12:15:46 : main.limit_compare                                      [ pass ]         3535
25 Feb 2011 12:15:48 : main.many_indexes                                       [ pass ]         1725
25 Feb 2011 12:15:56 : main.optimizer_subquery                                 [ pass ]         8659
25 Feb 2011 12:16:03 : main.outer_join                                         [ pass ]         6464
25 Feb 2011 12:16:09 : main.outer_join_portable                                [ pass ]         6535
25 Feb 2011 12:16:12 : main.repeatable_read                                    [ pass ]         2906
25 Feb 2011 12:17:42 : main.select_stability_validator                         [ pass ]        90206
25 Feb 2011 12:17:47 : main.subquery                                           [ pass ]         4344
25 Feb 2011 12:19:29 : main.subquery_semijoin                                  [ pass ]       102766
25 Feb 2011 12:19:39 : main.subquery_semijoin_nested                           [ pass ]         9195
25 Feb 2011 12:19:42 : main.varchar                                            [ pass ]         3225
25 Feb 2011 12:19:42 : ================================================================================
25 Feb 2011 12:19:42 INFO: Test execution complete in 286 seconds
25 Feb 2011 12:19:42 INFO: Summary report:
25 Feb 2011 12:19:42 INFO: Executed 18/18 test cases, 100.00 percent
25 Feb 2011 12:19:42 INFO: STATUS: PASS, 18/18 test cases, 100.00 percent executed
25 Feb 2011 12:19:42 INFO: Spent 286 / 286 seconds on: TEST(s)
25 Feb 2011 12:19:42 INFO: Test execution complete
25 Feb 2011 12:19:42 INFO: Stopping all running servers…

Each testcase is a simple .cnf file that defines a few crucial elements:

[test_info]
comment = starts up a master-slave setup.  Still needs some validation work
[test_command]
command = ./gentest.pl –gendata=conf/drizzle/drizzle.zz –grammar=conf/drizzle/optimizer_subquery_drizzle.yy –queries=10 –threads=1
[test_servers]
servers = [[–innodb.replication-log=true],[–plugin-add=slave –slave.config-file=$MASTER_SERVER_SLAVE_CONFIG]]

The test_command is the command line that is passed to the randgen tool itself.  We have created most of these tests and have put together the command lines that serve our testing needs.  By organizing our tests in this fashion, it is now easy for anyone to use the tool to perform meaningful tests.

The test_servers section is interesting – it is a simple list of python lists.  Each sublist contains a set of server options that a server will need.  Servers are started in-order with the first defined server being considered the ‘master_server’.  As we can see in the example to test the new slave plugin, the files are amazingly simple and clean.

Some other added benefits are that dbqp’s –valgrind and –gdb options work the same with randgen tests as they would with drizzletest cases.  Addtionally, we will be able to incorporate the code coverage provided by the randgen in our gcov reports.

One final little feature worth noting is the –gendata option.

The randgen provides a feature called the random data generator / gendata.pl
By passing it a configuration file that defines a set of test tables (see the forge documentation for further details), it will connect to a server, create the test tables, and populate them according to specifications.

While I have other plans for this, one cool use is for ad-hoc testing.  By passing a simple command line + the use of –start-and-exit, one can have a nicely populated test database to play with:

./dbqp –mode=randgen –randgen-path=/home/user/repos/randgen –gendata=/home/user/repos/randgen/conf/drizzle/drizzle.zz –start-and-exit
Setting –no-secure-file-priv=True for randgen mode…
<snip>
25 Feb 2011 13:28:23 INFO: Using testing mode: randgen
25 Feb 2011 13:28:23 INFO: Processing test suites…
25 Feb 2011 13:28:23 INFO: Found 18 test(s) for execution
25 Feb 2011 13:28:23 INFO: Creating 1 testbot(s)
25 Feb 2011 13:28:23 INFO: Taking clean db snapshot…
25 Feb 2011 13:28:23 INFO: testbot0 server:
25 Feb 2011 13:28:23 INFO: NAME: server0
25 Feb 2011 13:28:23 INFO: MASTER_PORT: 9306
25 Feb 2011 13:28:23 INFO: DRIZZLE_TCP_PORT: 9307
25 Feb 2011 13:28:23 INFO: MC_PORT: 9308
25 Feb 2011 13:28:23 INFO: PBMS_PORT: 9309
25 Feb 2011 13:28:23 INFO: RABBITMQ_NODE_PORT: 9310
25 Feb 2011 13:28:23 INFO: VARDIR: drizzle/tests/workdir/testbot0/server0/var
25 Feb 2011 13:28:23 INFO: STATUS: 1
# 2011-02-25T13:28:23 Default schema: test
# 2011-02-25T13:28:23 Executor initialized, id GenTest::Executor::Drizzle 2011.02.2198 ()
# 2011-02-25T13:28:23 # Creating Drizzle table: test.A; engine: ; rows: 0 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.B; engine: ; rows: 0 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.C; engine: ; rows: 1 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.D; engine: ; rows: 1 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.AA; engine: ; rows: 10 .
# 2011-02-25T13:28:23 # Creating Drizzle table: test.BB; engine: ; rows: 10 .
# 2011-02-25T13:28:24 # Creating Drizzle table: test.CC; engine: ; rows: 100 .
# 2011-02-25T13:28:24 # Creating Drizzle table: test.DD; engine: ; rows: 100 .
25 Feb 2011 13:28:24 INFO: User specified –start-and-exit.  dbqp.py exiting and leaving servers running…
../client/drizzle -uroot -p9306 test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 3
Connection protocol: mysql
Server version: 2011.02.2198 Source distribution (dbqp_randgen_updates)
Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.
drizzle> show tables;
+—————-+
| Tables_in_test |
+—————-+
| A              |
| AA             |
| B              |
| BB             |
| C              |
| CC             |
| D              |
| DD             |
+—————-+
8 rows in set (0.001614 sec)
drizzle> select count(*) from dd;
+———-+
| count(*) |
+———-+
|      100 |
+———-+
1 row in set (0.000614 sec)

Please give it a try if you feel so inclined.  We will be working to integrate this better into our build and test systems (Jenkins / make targets / etc).  And remember ./dbqp –mode=cleanup for quick and easy cleanup when you are done playing! ; )

Posted on by

Drizzle’s slave plugin is working!

These are exciting times for the Drizzle team.  We just released our first RC and things are finally coming together into some awesome new features.  I’m excited to bring you latest news from the replication front:

Where to begin?  Well, many moons ago, Brian sent David Shrewsbury and myself out on the task of making the transaction_log plugin rock solid.  This plugin provides a file-based log that captures the server’s state via protobuf messages.  After much blood, sweat, and tears (and *many* bugs), we accomplished our task with *plenty* of help from everyone on the team.  With this task accomplished, we could say that any replication solution using the log would have an accurate representation of the server’s state.  However, this was a long way from actually replicating server-server.

During this time, we were also working on storing the transaction log in an innodb table rather than a file.  Initial sysbench tests show a significant performance gain using this code versus the file based log.  Thanks to the initial work on the file-based log, getting this code up and running wasn’t too painful and it also passes all of our tests with flying colors.  Special mention should be given to Joe Daly, Stewart Smith, and Andrew Hutchings for hacking on this.

Having the transaction log in a database table provides other advantages beyond speed (such as easy, standardized access by other servers), but I’ll leave that to the hackers to discuss (I’d really recommend catching Dave’s UC talk if you are interested!).  The gist is that it has allowed the amazing Mr. Shrewsbury to cook up the slave plugin!

This is a plugin that allows a server to replicate from another server that is using the innodb-trx log.  It is amazingly simple:
master options:  –innodb.replication-log=true
slave options: –plugin-add=slave –slave.config-file=XYZ

The config file may contain the following options in option=value format:
master-host – hostname/ip of the master host
master-port – port used by the master server
master-user – username
master-pass – password
max-reconnects – max # of reconnect attempts if the master disappears
seconds-between-reconnects – how long to wait between reconnect attempts

The code hasn’t yet been merged to trunk, but can be checked out from lp:~dshrews/drizzle/slave
Currently, the plugin is able to replicate *anything* we throw at the master, which is HUGE!

Additionally, our experimental test-runner, dbqp, is sporting randgen integration!  I’ll write more about this in an upcoming post, but I mention it here as you can use the new randgen-mode + –start-and-exit to have yourself a handy-dandy replication setup for ad-hoc testing.  Observe:

./dbqp –mode=randgen –start-and-exit –suite=slave_plugin
Setting –no-secure-file-priv=True for randgen mode…
<snip>
24 Feb 2011 12:25:11 INFO: Using testing mode: randgen
24 Feb 2011 12:25:11 INFO: Processing test suites…
24 Feb 2011 12:25:11 INFO: Found 1 test(s) for execution
24 Feb 2011 12:25:11 INFO: Creating 1 testbot(s)
24 Feb 2011 12:25:11 INFO: Taking clean db snapshot…
24 Feb 2011 12:25:12 INFO: Taking clean db snapshot…
24 Feb 2011 12:25:13 INFO: testbot0 server:
24 Feb 2011 12:25:13 INFO: NAME: server0
24 Feb 2011 12:25:13 INFO: MASTER_PORT: 9316
24 Feb 2011 12:25:13 INFO: DRIZZLE_TCP_PORT: 9317
24 Feb 2011 12:25:13 INFO: MC_PORT: 9318
24 Feb 2011 12:25:13 INFO: PBMS_PORT: 9319
24 Feb 2011 12:25:13 INFO: RABBITMQ_NODE_PORT: 9320
24 Feb 2011 12:25:13 INFO: VARDIR: drizzle/tests/workdir/testbot0/server0/var
24 Feb 2011 12:25:13 INFO: STATUS: 1
24 Feb 2011 12:25:13 INFO: testbot0 server:
24 Feb 2011 12:25:13 INFO: NAME: server1
24 Feb 2011 12:25:13 INFO: MASTER_PORT: 9321
24 Feb 2011 12:25:13 INFO: DRIZZLE_TCP_PORT: 9322
24 Feb 2011 12:25:13 INFO: MC_PORT: 9323
24 Feb 2011 12:25:13 INFO: PBMS_PORT: 9324
24 Feb 2011 12:25:13 INFO: RABBITMQ_NODE_PORT: 9325
24 Feb 2011 12:25:13 INFO: VARDIR: drizzle/tests/workdir/testbot0/server1/var
24 Feb 2011 12:25:13 INFO: STATUS: 1
24 Feb 2011 12:25:13 INFO: User specified –start-and-exit.  dbqp.py exiting and leaving servers running…

We now have two servers running, a master on port 9316 and a slave on port 9321

user@mahmachine:~drizzle/tests$ ../client/drizzle -uroot -p9316 test

Reading table information for completion of table and column names

You can turn off this feature to get a quicker startup with -A

Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 4
Connection protocol: mysql
Server version: 2011.02.2197 Source distribution (drizzle)

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

drizzle> create table t1 (a int auto_increment not null, primary key(a));
Query OK, 0 rows affected (0.001652 sec)

drizzle> insert into t1 values (),(),();
Query OK, 3 rows affected (0.001182 sec)
Records: 3  Duplicates: 0  Warnings: 0

drizzle> select * from t1;
+—+
| a |
+—+
| 1 |
| 2 |
| 3 |
+—+
3 rows in set (0.000538 sec)

drizzle> exit
Bye
user@mahmachine:~drizzle/tests$ ../client/drizzle -uroot -p9321 test
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 82
Connection protocol: mysql
Server version: 2011.02.2197 Source distribution (drizzle)

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

drizzle> show tables;
+—————-+
| Tables_in_test |
+—————-+
| t1             |
+—————-+
1 row in set (0.000983 sec)

drizzle> select * from t1;
+—+
| a |
+—+
| 1 |
| 2 |
| 3 |
+—+
3 rows in set (0.00058 sec)

You will likely need to merge with trunk to use this feature in the slave branch, but the actual plugin should be merged soon.

When you are done playing, you can take advantage of a new quality of life feature – cleanup mode.
A pet peeve of mine is needing to clean up / shutdown any servers I may have started during testing.  With dbqp –mode=cleanup, the tool will now kill any server pids it detects in its workdir.  Nothing fancy, but useful, quick, and easy:

./dbqp –mode=cleanup
Setting –start-dirty=True for cleanup mode…
24 Feb 2011 12:32:55 INFO: Using Drizzle source tree:
24 Feb 2011 12:32:55 INFO: basedir: drizzle
24 Feb 2011 12:32:55 INFO: clientbindir: drizzle/client
24 Feb 2011 12:32:55 INFO: testdir: drizzle/tests
24 Feb 2011 12:32:55 INFO: server_version: 2011.02.2197
24 Feb 2011 12:32:55 INFO: server_compile_os: unknown-linux-gnu
24 Feb 2011 12:32:55 INFO: server_platform: x86_64
24 Feb 2011 12:32:55 INFO: server_comment: (Source distribution (drizzle))
24 Feb 2011 12:32:55 INFO: Using –start-dirty, not attempting to touch directories
24 Feb 2011 12:32:55 INFO: Using default-storage-engine: innodb
24 Feb 2011 12:32:55 INFO: Using testing mode: cleanup
24 Feb 2011 12:32:55 INFO: Killing pid 24416 from drizzle/tests/workdir/testbot0/server1/var/run/server1.pid
24 Feb 2011 12:32:55 INFO: Killing pid 24385 from drizzle/tests/workdir/testbot0/server0/var/run/server0.pid
24 Feb 2011 12:32:55 INFO: Stopping all running servers…

Please give it a whirl and help us make the code better by filing any bugs you detect!

Posted on by

Introducing dbqp.py aka project_steve_austin

So, what is all this dbqp / steve_austin mess?
Short answer – it is a nascent test runner.  Currently it duplicates the functionality of test-run.pl, but it is implemented in what I hope to be a more adaptable and reusable manner.  Please note that I said ‘duplicate’.  We haven’t touched test-run.pl or changed anything about how that tool works.  We don’t want to shoot ourselves in the foot by breaking a currently useful tool.

steve_austin…really?
Yes, it is test-run.pl, but better…stronger…faster ; )

dbqp?
database quality platform.  The idea is that this isn’t meant to just run a single type of test.  The code is intended to facilitate the execution of other tests.

Why?
Let me give you an example – any test of a database that I can think of requires starting up one or more servers (not much to test otherwise).  However, no project provides a standardized way of allocating and managing servers for tests (until now).  test-run.pl has a start-and-exit option (so does dbqp), but it seems crazy to have to call an entirely separate tool just to start a server (and such a method provides no management or cleanup – it is on the individual test to deal with that).  To run the randgen on Drizzle, one must start a server with ./test-run –start-and-exit, then call the randgen, then shutdown the server when finished : (  The situation is slightly better for MySQL / MariaDB, but the ./runall script the randgen uses still calls mysql-test-run.  Other tools define their own startup and management code, but such code has never seemed easily re-used to me.  This isn’t meant to be a knock on anyone or anything.  It is simply my belief that ‘things can be better‘ ™.

With dbqp, I have separated server allocation / management, system management (setting environment vars, finding files, etc), and test managment.  By structuring things in this way, we can define different testing modes.  Each mode uses some combination of adjusting / searching the system, starting and managing servers, doing things to the servers, then evaluating the effect of these ‘things’.  At present we only have a ‘dtr’ mode which runs our standard test suite, but further modes can be defined by simply writing a custom testManager (what does a testcase look like and how to package the test info) and testExecutor (how to execute / evaluate a testcase).  This will allow us to do things like create a ‘randgen’ mode which will facilitate executing our randgen transaction log tests (which currently involve too many manual steps outside of our automated build system) and do other cool things.

It should be noted that this is what the current test-run.pl basically does – it looks for testcases, starts the servers required by the test, packages the data and calls drizzletest.cc to actually execute the test (same for mysql-test-run.pl and mysqltest.cc).  It just makes sense to redo this in a way that will allow us to do the same thing for other types of testing tools : )

How can I play with it?
It is in the tree.  There is a make target – ‘make test-dbqp’.  I have tested it on Ubuntu and Mac OS X and it takes ~5 minutes to run our tests (all 551 of them!)
The manual page is here.

Please note that it is still in development.  It is reasonably solid and is beta quality, but test-run.pl is still in the tree for a reason ; )  All tests currently pass and one can run a variety of test / suite combinations as well as record tests.  Other utility features such as gdb and valgrind are still being polished.  Rely on test-run.pl for now, play with dbqp.
I’m also still adding and improving things as I write this – many, many thanks to Lee Bieber and Vijay Samuel for being early guinea pigs.

Anything else?
I’d like to give credit where credit is due and thank Monty Taylor for proposing the GSOC project that got me started on this path.  He gets at least part of the credit for any goodness this may bring (all responsibility for anything FAIL is on my shoulders).

There are some interesting things to come – we’re also moving on to the next phase of replication testing now that our trx log is rock solid

Posted on by

Drizzle’s transaction log is passing all tests!

In case you missed it here, we are very proud to announce that Drizzle’s transaction log is passing all of our tests.  For quite some time, David Shrewsbury, Stewart Smith, and Joe Daly have been putting a lot of love into the log code.  Please don’t be fooled by Dave’s praise of QA now that the storm has passed…you should have heard the names he called me and the things he plotted when we were rooting these bugs out ; )  However, now that there is a permanent record of his words, I’ll be reminding him about this post the next time my testing becomes a pain in his posterior and I feel him giving me the stink-eye in IRC (heheh)

With that said, we really have been putting tons of effort into making the log rock-solid.  This code will serve as the foundation for Drizzle replication and we can now be assured that any replication solutions will have a reliable log that will reflect the state of the server.  We have been beating up the code with the random query generator.  We have concocted several grammars that throw a variety of queries, in transactions and standalone, at the server.  After we have made the master do some work – we use a variety of connections and per-connection query counts, we produce SQL from the log, populate a validation server with the SQL, then compare Drizzledump files to ensure they match.  You should really check out the transaction_reader utility in drizzle/plugin/transaction_log/utilities – it allows a user to view the raw trx log contents, produce SQL from the log contents, and a few other neat tricks.  I’ll be blogging a bit more about how we used this for testing and troubleshooting very soon.

The transaction log worked well in most cases, the majority of our problems were in rooting out strange behavior around deadlocks and rolled back transactions.  I’d like to once again thank Stewart Smith, of the spork most flaming, for his valued assistance in finding these annoying bugs : )  If anyone wants to take a look at the bugs we’ve killed – you can check them out here.

We are far from being done – some of our next tasks include testing RabbitMQ, tweaking randgen tests to make certain we are totally crash-safe, and a few other things.  In the meantime, our randgen trx log tests run against every branch we intend to merge into trunk, so we’re keeping a close eye on making sure it stays solid.  We’ll keep you posted as our replication testing moves along.  Please keep trying Drizzle and helping us to improve it with your bug reports and feedback.

Posted on by

Testing status – Drizzledump migration

One of the key features of our first beta was drizzledump’s ability to migrate from a MySQL database.  For those who don’t know about this, the inimitable Andrew Hutchings enhanced drizzledump so that one could transfer data from MySQL directly to Drizzle without needing any intermediate files.  In this example, we connect to MySQL, running on 3306 and send the data in database my_mysql_schema, tables t1,t2, and t3 into our Drizzle server (port 4427).


$ drizzledump --compact --host=127.0.0.1 --port=3306 --destination-type=database --destination-host=localhost --destination-port=4427 --destination-user=root --destination-database=my_mysql_schema --user=root my_mysql_schema t1 t2 t3;

There are several ways of using this tool.  One could:

  • Transfer the data from all databases
    • $ drizzledump --compact --host=127.0.0.1 --port=3306 --destination-type=database --destination-host=localhost --destination-port=4427 --destination-user=root --all-databases --user=root  ;
  • Transfer data from schema ‘my_orig_schema’ on MySQL into schema ‘my_new_schema’ on Drizzle
    • $ drizzledump --compact --host=127.0.0.1 --port=3306 --destination-type=database --destination-host=localhost --destination-port=4427 --destination-user=root --destination-database=my_new_schema --user=root my_orig_schema
  • Transfer only the contents of specific schemas
    • $ drizzledump --compact --host=127.0.0.1 --port=3306 --destination-type=database --destination-host=localhost --destination-port=4427 --destination-user=root --databases --user=root my_schema1 my_schema3 my_schema5

The documentation can be found here and the blueprints for development and testing might be of interest as well.  The dev blueprint describes conversions and certain details that haven’t made it to the docs yet and the testing blueprint provides a list of bugs we’ve found. We found a number of minor, yet annoying bugs, but they have all been corrected at this time.

To return to the topic of testing – at first I had intended to use the randgen.  We already use it to test our ability to restore from drizzledump files.  While we did find some initial bugs with our tests, expressing certain things like indexes and foreign keys in a randgen grammar were proving problematic.

In the end, I created a new suite of tests for our standard test-runner.  The tests only have the following requirements:

  • You have a running MySQL server
  • You set two environment variables:
    • DRIZZLE_MYSQL_MIGRATE_TEST=1
    • DRIZZLE_MYSQL_MIGRATE_PORT= port where MySQL is running

Thanks to the utility dictionary plugin, we have the ability to SELECT environment variables and we use their presence to signal if we should run the test or not; if they aren’t set, the suite is skipped.  For example, we use this query in our checks:

SELECT VARIABLE_VALUE AS `TRUE` FROM DATA_DICTIONARY.ENVIRONMENTAL WHERE VARIABLE_NAME="DRIZZLE_MYSQL_MIGRATE_TEST";

When the tests run, they do the following:

  • populate the MySQL server with the relevant test-bed (tables + data)
  • call drizzledump to transfer the data from MySQL to Drizzle
  • perform various queries to ensure the transfer worked as expected (SHOW CREATE + SELECT * generally).

To run the tests:

./test-run --suite=mysql_migrate

You can expect to see the code in trunk shortly after this is posted.

We test every MySQL data type; we also test engines, indexes, foreign keys, and other setups.  While there are likely still some bugs left to be found, I believe that the tool should work smoothly for most users.  With that said, I would like to give a special thanks to Tim Soderstrom (aka Sweetums), MySQL DBA extraordinaire and Drizzle contributor.  He’s an early-adopter and helped us find a number of bugs while experimenting with migrating his own data to Drizzle.  Thanks again, Tim!

I hope that you’ll give the tool a spin and that you’ll write up a bug report if something isn’t working.  Questions and whatnot are always welcome on IRC and the mailing lists too : )

Posted on by

Sphinx documentation for Drizzle’s test-runner now available

In case you missed it, Drizzle is now using Sphinx to produce our documentation.  If you have sphinx installed (version 1+), you can generate them yourself with `make html`.  It is easy to work with (it’s Python, after all) and creates some very nice looking docs.

For those of you familiar with MySQL, test-run is similar to mysql-test-run, but with some adjustments for Drizzle.  It allows a user to run the test suite to ensure the system is performing correctly.  You can view the code coverage we achieve here.

One of the most important things people can do to help us move Drizzle from beta to GA is to try it out.  We do test very heavily, but extra sets of eyes are always helpful.  Let us know if things are broken or if you have thoughts on how things could work better; we welcome the feedback.

In the future, I intend to expand on the testing documents to include writing test cases and documenting the language features that are available in test-run.  Additionally, I will be writing up docs on how to use the randgen with Drizzle.  Please let us know via the mailing list / IRC / whatever if you have any specific information you’d like to see documented.

Also, if you are interested in contributing, but don’t necessarily want to hack on the code, I encourage you to tinker with the documentation – we are more than happy to accept patches : )  You can find the source files in drizzle/docs.