Tag Archives: NuoDB

NuoDB has new (web-based) management console

I don’t know whether you noticed, NuoDB is now in RC stage. If you’re following NuoDB long enough you might remember old console (you can check some images in my older post). It was a Java based application. Simple and it was not nice. Now it’s different. It’s web based. So it’s easier to connect to it. You can even connect to it from your phone (if it’s powerful enough to handle the JavaScript). It’s way nicer, nothing fancy (but I’m not expecting to be super cool, it’s basically admin panel for database, right?) and mainly, it has tons of new features.

After you log in, you see (initially) basic screen about your (now) so called domain.

Probably first thing you’ll try is starting new database. Simple wizard will allow you to do this. Here’s final screen before starting it up.

You can then see how is your database doing. Transaction nodes, storage nodes, memory etc.

You can even see some graphs about what’s going on (it’s live updated).

Very important for your database is keeping everything smooth and cleanly running. And if something goes wrong (or actually seems to be going wrong), act. In the new console you can set up alerts based on different events or metrics and browse these for investigation, for example.

You know I said I’m considering it being admin panel for database. Plain simple is OK for me. If I’m using it often enough my muscle memory learns the moves and I can get my information quickly whenever it is. Sometimes it takes little it longer, but hopefully we’re not doing this admin stuff often. But the new web console allows you customize some views to your needs, hence you can have all your information in front of you on few screens. You can show different views, organize these or collapse some. Some views are even parametrized, i.e. by database.

I like the new console. With the old one I was more happy with the command line and doing everything manually. With the new one I’m doing more and more stuff in it, especially if trying something.

I know I promised blog post about using NuoDB in cloud (Amazon EC2/Amazon S3), but still busy to prepare it completely. :\

Scaling with NuoDB

I know, there was a small gap between NuoDB related posts. I was little bit busy. But I’m now going to change it. In today’s post we’ll try to scale our chorus both in transactions and I/O throughput.

As you maybe remember from the first post, in NuoDB you have transaction and archive nodes. More transaction nodes means more queries/commands processed (yes, for badly designed application there might be other bottlenecks). And similarly with archive nodes that are in charge of persisting data to some storage and fetching data for mainly transaction nodes.

Simple. You already know how to start the agent and nodes from previous posts, but I think you will agree, that typing parameters on command line and keeping track what’s running where (and whether it’s running) is hard. This is where the NuoDB Console enters the game. It’s a simple application that shows you how your NuoDB world is looking.

Have a look at this picture (yeah, test machine is Windows XP 8-)).

You can see there’s one broker running on my ‘test’ machine. There are two choruses, with bunch of nodes. In this picture everything is on same machine, but in real environment you would probably have it across more machine. In ‘test‘ chorus is one archive node being started (the grayed one), currently not 100% ready and in ‘test2‘ chorus there is on the other hand some problem with one transaction node. For any problem you can check the log to see what’s wrong. From the same interface, using wizard, add nodes. Better than typing the commands again and again.

Computers running agents are being discovered automatically, but you can always add some manually if you know address, port and of course password.

This a great tool to start playing what happens when you kill (forcibly, that’s valid scenarios) some node and whether the chorus survives and all data are safe. My favorite scenario is one transaction node, two archive. Kill one archive node and continue inserting/querying data. Bring killed one back online and kill the other one. If you’re slow, the data will be synchronized, if not you’re screwed. :) BTW insert into table select * from table works so you can easily generated big results and long running commands.

And if I’m talking about playing, there’s also one program, that’s great for playing and seeing what’s going on. In samples\flights you can find application that allows you to put load to your setup. The bottom line is that you can play with number of clients (threads), also percent of updates and reconnect timeout. On the other hand in console you can try adding and removing nodes and see the system throughput. Whether it scales properly or whether you have some bottleneck there.


Wanna have more computing power? Add more machines with transaction nodes. Wanna be sure your data are safe in case of storage failure? Add more archive nodes. Also take into account, that each archive node holds complete data, not only some slices. So you should have enough free space available.

In next post we’ll deploy NuoDB into cloud (AWS) and use Amazon S3 as a storage for archive nodes (and as S3 in virtually unlimited, you don’t have to care about free space as mentioned above). Maybe I’ll also try flights demo in some medium size setup. ;)

Using NuoDB from .NET

In previous post I introduced you NuoDB, the so-called NewSQL database. On this foundation we’ll build today. Simple .NET/C# application to access the data, because if I can’t program it why bother with it, right? :)

NuoDB has currently JDBC and ODBC drivers in the box. Other drivers are created by community. There’s also .NET driver being created but it’s based on C++ API implementation not a fully managed implementation.

OK, so for now we’ll try the ODBC as it’s in the box. If you installed NuoDB, the driver was installed to your system already. If you’re going to deploy your application you should install it yourself. Whole ODBC is based on DSNs. You create one in advance and application the uses this one, with minimal knowledge advance how to connect to data source etc. Go to Control Panel > Administrative Tools > Data Sources (ODBC) and you can create DSN there. In .NET world we’re used to use connection strings to describe our data source, so instead of creating DSN manually, I’ll create it in application [1], just for my comfort.

The application is pretty simple, just to try something. Because NuoDB supports standard SQL, you can use constructs you’re familiar with, like CREATE TABLE etc.

using System;
using System.Data;
using System.Data.Odbc;
using System.Linq;

namespace NuoDB
{
	class Program
	{
		[System.Runtime.InteropServices.DllImport("ODBCCP32.dll")]
		static extern bool SQLConfigDataSource(IntPtr parent, int request, string driver, string attributes);
		
		static bool HandleDSN(bool remove)
		{
			return SQLConfigDataSource(IntPtr.Zero, remove ? /* ODBC_REMOVE_DSN */ 3 : /* ODBC_ADD_DSN */ 1, 
			     "NuoDB ODBC Driver\0",
			     "DSN=TestChorusDSN\0UID=admin\0PWD=admin\0Database=TestChorus@localhost\0");

		}

		static void Main(string[] args)
		{
			const string Separator = "\t|\t";

			Console.WriteLine("Adding DSN...");
			HandleDSN(remove: false);

			var csb = new OdbcConnectionStringBuilder();
			csb.Dsn = "TestChorusDSN";
			using (var conn = new OdbcConnection(csb.ToString()))
			{
				Console.WriteLine("Opening connection...");
				conn.Open();
				Console.WriteLine("Starting transaction...");
				using (var tx = conn.BeginTransaction())
				{
					string tableName = string.Format("user.test{0}", DateTime.UtcNow.Ticks);
					using (var cmd = conn.CreateCommand())
					{
						cmd.Transaction = tx;
						cmd.CommandText = string.Format("create table {0} (id int primary key, foobar string not null)", tableName);
						Console.WriteLine("Creating table...");
						cmd.ExecuteNonQuery();
					}
					using (var cmd = conn.CreateCommand())
					{
						cmd.Transaction = tx;
						cmd.CommandText = string.Format("insert into {0}(id, foobar) values (?, ?)", tableName);
						cmd.Prepare();
						IDbDataParameter parameter;
						for (var i = 0; i < 10; i++)
						{
							cmd.Parameters.Clear();

							parameter = cmd.CreateParameter();
							parameter.Value = i;
							cmd.Parameters.Add(parameter);
							
							parameter = cmd.CreateParameter();
							parameter.Value = string.Format("value of {0}", i);
							cmd.Parameters.Add(parameter);

							Console.WriteLine("Inserting...");
							cmd.ExecuteNonQuery();
						}
					}
					using (var cmd = conn.CreateCommand())
					{
						cmd.Transaction = tx;
						cmd.CommandText = string.Format("select * from {0}", tableName);
						Console.WriteLine("Reading data...");
						using (var reader = cmd.ExecuteReader())
						{
							Console.WriteLine(string.Join(Separator, Enumerable.Range(0, reader.FieldCount).Select(x => reader.GetName(x))));
							while (reader.Read())
							{
								var values = new object[reader.FieldCount];
								reader.GetValues(values);
								Console.WriteLine(string.Join(Separator, values));
							}
						}
					}
					Console.WriteLine("Committing...");
					tx.Commit();
				}
			}
			
			Console.WriteLine("Removing DSN...");
			HandleDSN(remove: true);
		}
		
	}
}

First I create DSN for my application (using ugly raw method), using same chorus set up like in previous post. The driver is called NuoDB ODBC Driver. Then I just specify username (UID), password (PWD) and database (Database) in form <chorus>@<location>, same as if you’re using nuosql. This DSN is also removed at the end.

Rest is simple .NET/C# code, same as any other ADO.NET code. Connection, transaction (if explicitly needed), command(s)…

I create table (hence DDL and ExecuteNonQuery works fine) with some magic 8-) name. Then some inserts with parameters (hence parameters, Prepare and ExecuteNonQuery with DML works) and finally select (hence basic stuff around ExecuteReader and IDataReader works).

Considering, I can store data in structured form in tables, use SQL and still be able to scale out easily, I think it’s neat. What could be even better? Having ADO.NET driver (preferable fully managed) and Entity Framework (aka LINQ) support. Maybe in the future…

What’s next? We’ll explore some failure and scaling scenarios.

[1]: http://www.codeproject.com/Articles/7652/Dynamically-adding-DSN-names

NuoDB – starting with “NewSQL” database

NoSQL is everywhere around us. And there’s a new (well, it’s here for a while, but still in beta stages of development) kind, it’s called NuoDB. To be precise, NuoDB positions itself as NewSQL. It claims to be 100% SQL, 100% ACID and 100% elastically scalable. I was following it since early stages (original name was NimbusDB, if somebody remembers) and I’d like to show it to you quickly. In follow up posts I’ll focus on using it from (mainly) .NET and some cloud scenarios (Amazon AWS, Windows Azure).

NuoDB is based on concept of chorus (aka cluster). Chorus is set of archive and transaction nodes (more about these later). One computer can be part of many choruses. Transaction node in chorus is node processing SQL transactions. If you add more transaction nodes into chorus, you basically scale it up (you can process more transactions, in, hopefully, less time). The archive nodes are nodes used for writing data into storage (just a side note, NuoDB has support for Amazon S3 without EBS). If you add more archive nodes into chorus you’re more tolerant to storage failures. There’s also “special” process (or more), called broker that manages the chorus and through it the first interaction with database occurs (then you’re talking to transaction nodes). And all this is managed through “agent” process.

Pretty straightforward, isn’t it (sure, devil is in details)? So let’s get our hands dirty. All these examples are on Windows, but you can select from variety of platform while downloading NuoDB (and of course, you can mix platforms).

>start java.exe -jar bin\nuoagent.jar --broker --password FooBar

>start bin\nuodb.exe --chorus TestChorus --password TestPwd --dba-user admin --dba-password admin

>start bin\nuodb.exe --chorus TestChorus --password TestPwd --archive "C:\Users\Jiri\Desktop\NuoDBStorage" --initialize

The first command starts broker. Second starts transaction node in TestChorus with initial user/pwd admin/admin. And last command starts archive node, the data will be stored in C:\Users\Jiri\Desktop\NuoDBStorage directory. The --initialize switch is used only first time, to, well, initialize the storage itself. That’s it, minimal set up to start playing with NuoDB. Now we can connect to it and start playing with SQL.

>bin\nuosql.exe TestChorus@localhost --user admin --password admin
SQL> create table Contacts (ID int primary key, FirstName string, LastName string, Phone string, Street string, HouseNumber int, City string);
SQL> commit;
SQL> insert into Contacts values (1, 'Jiri', 'Cincura', '+420123456789', 'RRR Street', 6, 'RRR City');
SQL> commit;
SQL> select * from Contacts;

 ID  FIRSTNAME  LASTNAME      PHONE       STREET   HOUSENUMBER    CITY
 --- ---------- --------- ------------- ---------- ------------ --------

  1     Jiri     Cincura  +420123456789 RRR Street      6       RRR City

SQL>

Easy, right? So get grab it and play with it. Next time we’ll try some failure scenarios and we’ll explore how to connect from .NET and some other environments. Feel free to ask if you’re interested in something.