Horizontal Partitioning 1

When you want to speed up the physical reads on a SQL Server, you have two options: faster disks, or more disks. But only once you've determined that physical reads are a (potential) bottleneck do you get to make the choice.

If using faster disks is not enough to get your system back to an acceptable level of performance, then there's still the option of adding more disks. Merely adding disks won't - in itself - speed anything up. To realize performance gains you need to restructure your data. SQL Server allows you to partition tables, and also indexes. But as usual, the devil resides in the details.

Partition functions can only take one parameter. This means that the partition in which each row resides is determined by the value of just one column value in that row.

If your original table had a clustered index, you'll probably want to keep it. However, this has a big consequence: you will need to make the partition function congruent with the clustered index. SQL Server will complain if you leave the partition column out of the clustered index "Partition columns for a unique index must be a subset of the index key". It gets worse if you want a composite clustered index - you should be aware that in some cases SQL Server appears to store data internally sorted first by the partition column, and then by any other columns in the composite clustered index. If your original table was sorted by Col_1,Col_2 and you choose Col_2 as your partition column, then your table may be sorted internally by Col_2,Col_1. Actually, it's not this straightforward: I need some time to figure this out; it will be the subject of a later post.

Think Big

I was playing around with my new installation of Windows 7 today, frustrated that I couldn't install SQL Server 2008 Management Studio Express (not yet supported), but determined to have fun. I was also playing around with 64-bit Windows Server 2008 running SQL Server 2008 on a machine with 8GB of RAM. I thought it would be fun to create a table with 2^32 rows, and I was right. If you run SELECT COUNT(*) FROM #Big on a table of this size, you're shown an error message "Arithmetic overflow error converting expression to data type int." No fear though, just use the COUNT_BIG() function instead of COUNT(). Bingo: 4294967296!

Realistic Scalability

Everybody seems to focus on adding processors and memory (or complete nodes) when they talk about scalability, but not a lot of mention is made about adding new people to manage the systems. True scalability should, in my opinion, include factors like the cost of human labour. You've designed a new system: great! It runs with only 20 errors a day on 1000 engines: super! It's linearly scalable, so your boss buys another 1000 engines: unfortunately, this means an extra 20 errors per day; this could mean another person needs to be added to the support team!

Escribir en español

I'm using Ubuntu 8.10 and learning to speak Spanish, so I wanted to know how to type the accented characters and inverted exclamation and question marks. As it turns out: it's pretty easy, even with a generic UK keyboard. Follow these easy steps to do it yourself:

System > Preferences > Keyboard

On the Layouts tab, click the "Other Options..." button, then expand the "Compose key position" element. Select "Right Alt is Compose." Close the Keyboard Layout Options dialog and focus the cursor on the "Type to test settings" box.

For á, é, í, ó and ú:
Press and release Right-Alt, then press and release @', then press and release the vowel key.

For the upper case Á, É, Í, Ó and Ú:
Press and release Right-Alt, then press and release @', then hold shift while pressing and releasing the vowel key.

For ñ:
Press and release Right-Alt, then press and release ~#, then press and release n.

For the upper case Ñ:
Press and release Right-Alt, then press and release ~#, then hold shift while pressing and releasing N.

For ¿:
Press and release Right-Alt, then hold shift while pressing and releasing the ?/ key twice.

For ¡:
Press and release Right-Alt, then hold shift while pressing and releasing the !1 key twice.

IIS and Process Orphaning

IIS has some very neat features. Consider process orphaning: your application has become unresponsive... IIS can take it out of the application pool so that it receives no further requests, and can automatically attach a debugger or write a full memory dump for you to examine later. And while all this is happening, it will have brought up a new worker to seamlessly service all other inbound requests. All in the name of reliability.

Isolation / Immutability / Synchronization

If there's any one recipe for software runtime scalability, it's this: (IIS)^2. It's no secret I'm a fan of Internet Information Services, but in today's post it plays the role of host to our application, and my focus instead is on the underlying application's design.

Isolation:

Prevent state from being shared, allowing us to read and write to our hearts' content. Shared Nothing (SN) architectures are the ultimate realization of isolation. The definition of the isolation boundaries within an application might become contentious, and someone will need to be responsible for owning each isolated domain.

Immutability:

Prevent state from being modified - we can have as many readers as we want. In reality though, immutability by code contract is not as straightforward as simply not providing setter methods. In distributed environments especially, the .NET runtime must (de)serialize objects, which requires default constructors and public get/set accessors. We could choose to pass restrictive interface references instead, but this would force the (un)boxing of structs. Possibly, the application of custom attributes to state modifying methods/properties could be inspected statically by a tool like FxCop to prevent accidental object mutation. Objects do require mutation at times to be useful.

Synchronization:

Mark critical sections where locks must be acquired before reading or writing. This is the typical style of programming that I've seen in Java and C# and doesn't easily lend itself to distributed programming. Also, while waiting, valuable resources are often wasted. It tends not to scale as well.

The best solution to a large distributed computing problem will undoubtably be a combination of the three above problems. Isolated tasks are likely candidates for physical distribution as they will scale horizontally. Immutability is hard to enforce completely and synchronization implies necessary waiting for other tasks. For these reasons, I give top priority to isolation, and let the other two duke it out as I see fit on the day.

Without Intellisense, I'm Nothing

Imagine you're given a task: write the code for a program in a hitherto unknown DSL, embedding that code within a hitherto unknown markup language. What would be your tool of choice? Without any easily understood reference documentation, it's not going to be easy. Without the visual clues that you're missing a quote (or your string contains an extra quote) or that your function call doesn't exist or has the wrong arguments, you're going to have to run the program just to debug it. If that running process involved multiple clicks and typing followed by a short wait, you're going to get frustrated and probably won't deliver the program on time. Intellisense is great. You can write Javascript inside an HTML page, and Visual Studio will squiggle under everything you've done wrong!

Hello Ruby

def reverse(value)
  if value.length > 1 then 
    value[value.length-1] + reverse(value[0,value.length-1]) 
  else 
    value[0] 
  end
end

The string formatting functions of Ruby caught my interest, but I haven't checked if any equivalents exist in Python.

irb> "#{reverse 'Hello, World!'}"
=> "!dlroW ,olleH"

Synchronisation... is... slooooow...

a.k.a an argument for Shared Nothing. Continuing my !random post, I decided to see how long it would take to build up a cumulative normal distribution using the values generated by System.Random.Next(). I used and timed the following configurations:
1) Four threads on four cores sharing a Random (with thread synchronisation): 180 seconds.
2) One thread on one core with its own Random: 63 seconds
3) Four threads on four cores, each with their own Random (sharing nothing): 17 seconds.
The lesson here is that synchronisation is slow. So slow in this case, that it was actually faster to single-thread the application. However, when each thread was given its own object that it didn't have to share with the others, the speed increase was dramatic. Not only was it massively faster, but I'd put my money on it scaling pretty well with an 8, 16, 32 or even 64 way server.

Out of curiosity, I also tried this configuration:
4) Four threads on four cores, each with their own Random, locking it unnecessarily: 39 seconds.
I found it interesting that even uncontended locks could halve the speed of my algorithm.

The moral of the story is to think about your locking strategy (hint: avoid putting yourself in the position where you need one) when looking to parallelize tasks.

!random

System.Random's Next() method isn't guaranteed to be thread-safe, so if you start calling it from multiple threads at the same time, you're likely to end up with very little entropy indeed! I cooked up a static class, with a static System.Random field, and set 4 threads in motion calling Next() continuously. Each thread got its own core to run on, and very soon the only "random" number returned from Next() was 0.0 - the object had been well and truly corrupted. At this point I needed to choose: a single System.Random protected by lock() statements, or multiple System.Random objects. If I chose the single route (why?) all the synchronization would slow me down, and I'd end up not using each core to its fullest potential. If I chose the multiple route (only as many System.Random objects as there were threads), I would need to seed each one with a different value, otherwise they could - if created at the same time - return the same series across more than one System.Random object.

Interestingly, if I called the overloaded version of Next(int min, int max), soon the only return values would be min.

Buffer vs. Array

To clear up any misconceptions, System.Array.Copy is not the same as System.Buffer.BlockCopy unless you're operating on arrays of System.Byte. The Buffer class copies bytes from one array to another. If you have an array of System.Int32 (4 bytes each) and you copy 3 bytes (not items) from your source to your destination array, you will just get the first 3 bytes of your 4 byte System.Int32. Depending on the endian-ness of your system, this could give you different results. Also, the System.Buffer class only works on primitives. Not C# primitives (which include strings) and not even derivatives of System.ValueType (e.g. enums and your own structs). Clearly it can't work on reference types safely (imagine just copying 3 of the 4 bytes from one object reference to another) but I would have expected it to work with enumerations (essentially ints).

Hello Python

>>> def reverse(value):
 if len(value) > 1:
  return value[len(value) - 1] + reverse(value[:len(value) - 1])
 else:
  return value[0]

 
>>> print reverse('!dlroW ,olleH')
Hello, World!

System.OutOfMemoryException Part 4

And then there were 2. I'm talking about code paths that lead to an OutOfMemoryException being thrown by the CLR.

#1 is the standard "You've run out of contiguous virtual memory, sonny boy!", which is pretty easy to do in a 32-bit process. With the advent of 64-bit operating systems, you get a little more headroom. Actually, a lot more headroom: precicely 2^32 times as much as you had to start with! But, due to limitations built into the CLR, you're no further from an OutOfMemoryException...

#2 is when you try and allocate more than 2GiB in a single object. In the following example, it's the +1 that pushes you over the edge:
new System.Byte[System.Int32.MaxValue + 1];
If you're using a different ValueType your mileage may vary:
new System.Int32[(Ssytem.Int32.MaxValue / sizeof(System.Int32.MaxValue)) + 1];
Or if you're on a 64-bit system:
new System.IntPtr[(System.Int32.MaxValue / sizeof(System.Int64)) + 1];

Many thanks to this blog for helping me in the right direction here.

Hopscotch in the 64-bit Minefield

So it's no secret I've been playing with virtualization, Windows, Linux, IA32 and amd64. Virtualization looks the part, but so does a 24-bit color depth 1280 x 1024 bitmap of a fake desktop. You can't do much with either.

Microsoft has given us WOW64, allowing us to run old x86 (IA32) software on new 64-bit operating systems. It's so seamless you forget you've got it: until you try and install ISA Server 2006.

Getting Windows Hyper-V Server up and running on a headless box... well, let's just say I'm not that patient. I eventually settled for Hyper-V role on a full-fat Windows Server 2008 Enterprise Edition install, with crash-carted DKM. It doesn't appear to run on anything except the 64-bit version, either. Even then, loads of caveats await:
gigabit network card? you'll have to use a ridiculously slow emulated "switch" instead.
multiple cpus? not!
scsi drives? think again.
usb? umm... for some reason nobody in the world has ever thought of plugging a usb device into a server. i was the first. i'm so proud!
Granted, all these restrictions are imposed on the non-MS (see ubuntu) and older guest operating systems like Windows XP, but isn't half the pitch behind virtualization about getting rid of old physical kit and consolidating servers?

Flex Builder 3? Oh yes. But not for Linux. No wait... there's a free alpha version but it's not compatible with 64-bit.

VMWare ESXi? Check your hardware list first.

Debug Enable!

All wireless functions on my O2 wireless box II appear to have died overnight. This is a problem for me because I use wireless all the time... to stream music to my Apple TV, to surf the Internet from my laptop, and play games on my iPod Touch. The wired ethernet is still happily routing packets between my LAN and the Internet. So I climbed up in my cupboard and pulled down the old Netgear DG834G - what a beautiful beast. It runs BusyBox, an embedded Linux operating system and you can connect to it using any telnet client. Just navigate to http://<netgear-router>/setup.cgi?todo=debug from your browser first, and you're A for Away - you can then telnet into it. Reboot the rooter when you're finished to disable the telnet server until next time.

Things that frustrated me:
There was no way to change the default gateway address handed out by either router. I suspect I could have done so with the Netgear box in time, but there is no text editor in the distribution.

To get my network back up and running would be difficult without the usual trance music coming through the Apple TV; keeping me focused. Still, I needed to:

1. put both routers onto the same subnet
   
2. keep DHCP running on the O2 box (so that the default gateway would remain the O2 box address)
   
3. turn off DHCP on the Netgear box (otherwise it would give out its own IP address as default gateway)
   
4. turn off the wireless interface on the O2 box for good
   

In a nutshell:
o2wirelessbox is now 192.168.1.254/24 with DHCP handing out addresses in the range 64-253
netgear is now 192.168.1.1/24 with disabled DHCP (I would have liked to use the range 2-63)

Gigabit Ethernet

... is actually quite fast. So fast - in fact - that the bottleneck on both PCs under my desk is currently the PCI bus into which the network cards are plugged. I had no idea that the bus ran at 33MHz and was only be able to transfer 32 bits per cycle (math says: 33M * 32b = 1056Mb). Todo: is this duplex?

There's a very useful FAQ regarding Gigabit Ethernet available here.

In a test with 4 network cards (2 in each machine) I saw the following:

w) 1A -> 2C (93Mb/s)
x) 1A -> 2D (902Mb/s)
y) 1B -> 2C (93Mb/s)
z) 1B -> 2D (294Mb/s)

1A: Gigabit embedded on motherboard (1000 link at switch)
1B: Gigabit on PCI bus (1000 link at switch)
2C: Fast embedded on motherboard ( 100 link at switch)
2D: Gigabit on PCI bus (1000 link at switch)

Machine 1 was sending large UDP datagrams. Machine 2 was not listening, it was just up so that ARP could get the MAC addresses of its adapters (without which, we could not send a datagram).

Interestingly:
tests w + y appeared to be throttled by the router as it managed a mixed 100/1000 route
test x was great and showed that an onboard gigabit controller can actually do its job
test z showed the PCI bus being the bottleneck, allowing 3x faster than Fast, but 3x slower than Gigabit.

Extellect Utilities

I've finally put a set of C# productivity classes on Google Code under the Apache 2.0 License. So, check 'em out, see if they make your work any easier, and let me know what you think.

Remoting, bindTo and IPAddress.Any

Just solved a fun problem. While running a .NET remoting server on my ubuntu box with multiple NICs I saw some strange behavior where the server would return an alternate IP address, and the client would attempt to re-connect (using a new Socket) to the new IP address. Problem being: the server was responding with its localhost address (127.0.1.1), which the client was resolving to its own loopback adapter. See the problem yet?

It turns out that in the absence of a specific binding, the server binds to IPAddress.Any. When a client attempts to connect, it's redirected to the server's bound address. Unless client and server are hosted on the same physical machine, there's really no point in ever using the loopback adaptor... which makes it a strange choice for default.

The solution:
Before you create and register your TcpServerChannel, you need to set some options.
IDictionary properties = new Hashtable();
properties["bindTo"] = "dotted.quad.ip.address";
properties["port"] = port;
IChannel serverChannel = new TcpServerChannel(properties, new BinaryServerFormatterSinkProvider());
RemotingConfiguration.RegisterWellKnownServiceType(typeof(Explorer), "Explorer.rem", WellKnownObjectMode.SingleCall);

Voila! 'appiness should ensue...

PS. If for some reason the dotted quad doesn't appeal to your particular situation (e.g. load balancing), you can set two other properties instead:
properties["machineName"] = "load.balanced.server.name";
properties["useIpAddress"] = false;

PPS. I think the client will make always two socket connections, A + B. A is used at the start to do some initialization and get the address for connection B. B is used for meat and bones of the operation, and finally A is used just before they're both closed.

Goodput - Season Finale

I thought I'd take a look at a couple of different (and by no means an exhaustive list of) options for transferring a reasonably large file across a network. Over the past couple of days I tried sending a 700MB DivX using the .NET Remoting API (over both TCP and HTTP), the .NET Sockets API (over TCP), and finally using a mounted network share, reading the file as if it was local.

The table that follows shows the results of these tests:

I'd advocate taking pinch of salt when interpreting the numbers.

In general, remoting doesn't support the C# compiler generated closures it emits when it compiles an iterator block (e.g. the yield return keyword): quickly remedied by exposing the IEnumerator<T> as a remote MarshalByRefObject itself, wrapping the call to the iterator block. This gave us a nice looking (easy to read) interface, but will have increased the chattiness of the application, as every call to MoveNext() and Current would have required a network call. Further to this, the default SOAP serialization used with HTTP remoting doesn't support generic classes, so I had to write a non-generic version of my Streamable<T> class.

The performance of the HTTP/SOAP remoting was abysmal and there was very little gain by switching to a faster network. Even with what I suspect to be a massively chatty protocol (mine, not theirs), the bottleneck was probably somewhere else.

TCP remoting was next up. Under the covers it will have done all the marshalling/unmarshalling on a single TCP socket, but the chatty protocol (e.g. Current, MoveNext(), Current, MoveNext() etc.) probably let it down. TCP/Binary remoting's performance jumped 2.5x when given a 10x faster network, indicating some other bottleneck as it still used just 16% of the advertised available bandwidth.

CIFS was pretty quick, but not as quick as the System.Net.Sockets approach. Both used around 30% of the bandwidth on the Gigabit tests, indicating that some kind of parallelism might increase the utilization of the network link. An inverse-multiplexer could distribute the chunks evenly (round-robin) over 3 sockets sharing the same ethernet link, and a de-inverse-multiplexer (try saying that 10 times faster, after a couple of beers) could put them together.

Back on track...

Seeing as TCP/Binary remoting was the problem area that drove me to research this problem, I thought I'd spend a little more time trying to optimise it - without changing the algorithm/protocol/interface - by parameterizing the block size. The bigger the block size, the fewer times the network calls MoveNext() and get_Current have to be made, but the trade-off is that we have to deal with successively larger blocks of data.

What the numbers say: transmission rate is a bit of an upside down smile; at very low block sizes the algorithm is too chatty, at 4M it's at its peak, and beyond that something else becomes the bottleneck. At the 4M peak, the remote iteration invocations would only have been called 175 times, and the data transfer rate was 263Mb/s (roughly 89% of the observed CIFS' 296Mb/s).

Full Duplex

Simple english: if two computers are connected by a full duplex ethernet link, then they should be able to carry out two conversations with each other simultaneously. For example, imagine two computers named A and B with a 100Mb/s full-duplex connection linking them both. A starts "talking" at 100Mb/s and B "listens". B also starts "talking" at 100Mb/s and A "listens". The total data moving up and down the link 200Mb/s. That's full duplex, baby!

Only, in real life you don't get the full 100Mb/s in either direction. On my PC, I managed to get 91Mb/s in one direction and 61Mb/s in the other direction. If I stopped the 91Mb/s conversation (call it X), the 61Mb/s conversation (call it Y) would quickly use up the extra bandwidth, becoming a 91Mb/s conversation itself. As soon as I restarted X, it reclaimed its original 91Mb/s, and Y returned to its original 61Mb/s. Freaky.

Goodput - Part 2

So then I thought to myself, "Hey, you have two NICs in each machine. Why don't you try and get double the throughput?" Even though all my NICs are gigabit ethernet, my modem/router/switch is only capable of 10/100 (a gigabit switch is in the post, as I type). Yesterday's tests indicated that I was getting roughly 89Mb/s, so I'd be aiming for 178Mb/s with my current hardware setup. And a glorious (hypothetical) 1.78Gb/s when the parcel arrives from amazon.co.uk.

What would have to change? For starters, the server was binding one socket to System.Net.IPAddress.Any; we'd have to create two sockets and bind each one to its own IP address. Easy enough. The client would also have to connect to one of the the two new server IP addresses.

Wait a minute... there isn't any System.Net.Sockets option on the client side to specify which ethernet adapter to use. You only specify the remote IP address. Oh no! This means we could end up sending/receiving all the data through just one of the client's NICs. Luckily, you can modify the routing table so that all traffic to a particular subnet can be routed via a specific interface. I'm using ubuntu as the client, and my routing table looks like this, which indicates that eth0 would get all the traffic to my LAN:

Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.0 * 255.255.255.0 U 1 0 0 eth0
192.168.1.0 * 255.255.255.0 U 1 0 0 eth1

I want to add a static route, with higher precedence than the LAN subnet, using eth1 for all communication with the remote IP 192.168.0.73, leaving eth0 with the traffic for the rest of the 192.168.1.0/24 subnet. I type this command at the console:

sudo route add -net 192.168.1.73 netmask 255.255.255.255 dev eth1

Disaster averted. The routing table now looks like this, and I'm happy to say my diagnostics report that I'm getting around 170Mb/s with my new trick in place. It's not the 178Mb/s I was hoping for (I've lost about 4.5% on each connection), but it's still 190% of the original throughput.

Destination Gateway Genmask Flags Metric Ref Use Iface
192.168.1.73 * 255.255.255.255 UH 0 0 0 eth1
192.168.1.0 * 255.255.255.0 U 1 0 0 eth0
192.168.1.0 * 255.255.255.0 U 1 0 0 eth1

Throughput comparison reading a 700MB DivX file:
73Mb/s - Mounted network share using CIFS (although it also appeared to be caching the file on disk, incurring disk write penalties)
89Mb/s - 1x NIC using System.Net.Sockets
170Mb/s - 2x NIC using System.Net.Sockets

Goodput - Part 1

I've been interested recently in maximizing the application-level throughput of data across networks. The word I didn't know I was looking for - but found anyway - was goodput.

At first I tried streaming across a large DivX file. When I realised that I might be measuring disk seek time (doubtful, at the pitiful data rates I was achieving) I transitioned my tests to stream meaningless large byte arrays directly from memory (being careful not to invoke the wrath of the garbage collector, or any of the slow operating system-level memory functions).

What I noticed was that the application-level control of the stream was a big factor in slowing down the effective data transfer rate. In short, if my design of the logical/physical stream protocol was "bad", so would be the goodput.

Throughout the test, data was transmitted in chunks of varying sizes (from 4k to 128k). Firstly, just to see what it was like, I tried establishing a new System.Net.Socket connections for each chunk. Not good. This is why database connection pooling has really gained ground. It's expensive to create new connections. Next I tried a single connection where the client explicitly requested the next chunk. Also really bad. It was chatty like an office secretary, and got less done. So I tried a design I thought would be pretty good, where 1 request resulted in many chunks being returned. For some reason, I thought that prepending each chunk with its size would be a good idea. It was 360% better than the previous incarnations, but the extra size information was just repeating data that wasn't at all useful to the protocol I had devised: it was wasting bits and adding extra CPU load, and giving nothing in return; it had to go. Stripping these items from the stream resulted in an extra 3.6% of throughput.

Interestingly, I noticed that the choice of buffer size could drastically affect the goodput, especially when it was ((1024*128)+4) bytes. I expect this was something to do with alignment. It would be cool to do some more tests, looking for optimal buffer sizes.

LiveCycle Data Services / amd64 / ubuntu

I haven't tried out the alternative - BlazeDS - yet, but I've had the time of my life trying to get ubuntu to use all my RAM and both my monitors, and even more fun trying to get LCDS installed. Nothing has been exceptionally difficult, but I've noticed a lot of things just are not compatible. Knowing not only where you're going, but also the route to get there, can make a major difference in total time taken. That's why I document my "mistakes" - for next time. Next time...

1. Well, maybe next time I'll get Hyper-V Server 2008 to work. Even VMWare ESXi wasn't happy with my SCSI drives. Xen looked like more than I was prepared to bite off, nevermind chew.


2. Accessing any more than 3GB of physical RAM on an ubuntu install requires the amd64 package. Similar limits are imposed on other 32 bit operating systems. In this case I chose to upgrade to the 64 bit version than enable PAE.


3. to do anything useful in an unfamiliar operating system you'll want to use the desktop version.


4. the graphics driver and operating system seemed to fight like small children if the driver's not open source ... I'm just glad that nVidia have made the effort for a 64 bit driver so I can see my desktop in TwinView, rather than being forced to run the server version (see 3).


5. the Flash plugin that you can download from Adobe's website is - at time of writing - only compatible with i386 and not amd64. Problem's easily solved by using sudo apt-get install flashplugin-nonfree which was already set up inside Aptitude.


6. when you download a .bin file, you're expected to know you're going to chmod +x and execute it. However, in the case of lcds261-lin.bin you can't just do this inside a terminal from the CTRL-ALT-F7 session; you must run it from one of the CTRL-ALT-F{2..6} sessions. Else you'll get a Java error java.lang.UnsatisfiedLinkError: Can't load library: /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/xawt/libmawt.so. Amusingly, one of the lines in the stack trace read: at ZeroGay.a(DashoA10*..)


7. even if you asked the installer for a Tomcat install, don't expect your container to be started when installation is complete. Perhaps it's too much to expect a reasonable error message too. Anyway, I created a new service account user "tomcat" with sudo adduser tomcat and then gave it ownership of the (otherwise locked down) Tomcat install directory and started the server.


8. Actually check that the samples are working - front-to-back - at this point. First time around, I celebrated an early victory when I first saw the sample pages running inside Tomcat. There was a leeeettle bit more setup to do re: the sample database. The LCDS install ships with an HSQLDB database, which needs write permission on a bunch of directories. In the spirit of the previous step (and, perhaps, a pre-existing condition in which I derive pleasure from creating new service accounts) I created a new user called hsqldb with permission to create the .lck (lock) files. sudo adduser hsqldb and then /opt/lcds/sampledb$ sudo chown -R hsqldb .


9. LCDS is free under Adobe's developer license on more than 1 CPU (or even in production environments, but only on a single CPU). This is great for developers like me who can tinker at home without parting with hard cash. We can even demo our applications to clients for free. The people with the cheque books can keep them in their pockets until bigger test and production environments are commissioned.

System.OutOfMemoryException Part 3

It's not every day you come home and plug 8GB of physical memory into your home PC. Before now, I wasn't completely aware that there was even a limit to the amount of memory you could usefully install on a Windows XP machine. There are all sorts of hurdles it seems. Luckily, Windows Server 2008 x64 has no problem with addressing it all. Bonus. But can you imagine the size of the swap files you would start to rack up if you ran a bunch of memory hungry processes simultaneously. Processes only see virtual memory... Windows is smart about whether this memory is backed by physical memory pages, or pages allocated on a system file called the swap file. As we move to the wider 64-bit addresses, we're not going to get another System.OutOfMemoryException; instead we'll run out of disk space for the swap file or spend so much time thrashing that we'll want the small address space back!

Gen / Spec

I think it's common for vendors and consultancies to push their own technologies as solutions to the problems of clients. Even individual consultants are often bullish about the skills they've acquired. Together, these behaviours make it difficult (if not impossible) for the optimum solution to be found and implemented. Of course, an optimum solution isn't always what a client wants: consider delivering a sub-optimal solution that leaves your client better off than the nearest competitor. Still, I feel that recognising that an optimal solution does exist is an important step towards better architectures.

Adam Smith - the economist - wrote about the division of labour in his magnum opus, The Wealth of Nations. To paraphrase: he talks of a pin factory and the difference in productivity between two scenarios - sharing responsibilities across a group of employees, and assigning specific tasks to employees. There are many conclusions one can draw, but the one that shines out particularly to me is that performance gains can be made by choosing the right tool for the job (RTFJ).

Databases. Great for relational data storage and retrieval. They're designed to meet ACID requirements, and even with all the log shipping and replication in the world, don't scale quite as nicely as other technologies can/do. In my book, that would have been reason enough to keep business logic out of the database. However, certain cases might call for a gargantuan set of data to be worked on at once, and it might be prudent to "bring the computation to the data".

Grids and high performance computing. Great for compute intensive operations. However they're distributed by nature, and that generally makes things more difficult. They usually offer only a subset of the common operating system constructs we're used to - well conceptually, anyway. Spinning up a new thread locally is the "equivalent" of starting up a new process on a remote machine. Also there's the problem of moving data. (De)serialization is computationally intensive - optimisations can take the form of using shared metadata of common versions (e.g. .NET assemblies, and binary serialization) which bring new problems of managing versioning across the environment.

Whatever you're doing, always make "efficient" use of your CPU. Use asynchronous patterns for non-CPU tasks (e.g. waiting on I/O) using callbacks. Thread.Sleep() and spinning in a tight loop are generally evil (but I'm sure there exists a case where both are specifically wonderful).

Distribute only what you have to. If your constraint is virtual ("addressable") memory then it might be OK just to have multiple processes on the same machine with lots of physical memory, talking to each other via some non-network IPC mechanism.

Cache hits should be quick. Cache misses (generally) shouldn't result in multiple simultaneous requests to insert fresh data in the cache. Tricky bit is not making any "threads" wait while the cache data is inserted. That ties my previous point in with the next:

DRY. Don't repeat yourself. This goes for operations as well as for boilerplate copy and pasted code. If a cache can give you the result of an expensive operation you've already computed, for less cost, then consider caching. In-memory, disk, distributed and bespoke caches exist. Each will have a

Licensing Components in .NET - Part 2

I reckon the only reason LicFileLicenseProvider is part of the framework is to get the point across the licensing greenhorns. All of a sudden, brains start ticking: you can load the license from anywhere. You begin crafting nefarious schemes using public key cryptography. It's brilliantly academic. But something's wrong. It would be easier just to hack the code. Steal the intellectual property. Hey, maybe that's why Microsoft stopped at the LicFileLicenseProvider? Maybe, and here's a thought: maybe they should have.

Another crazy piece of the (even crazier) licensing puzzle: licenses.licx files and lc.exe.

lc.exe is a tool written in .NET by Microsoft, which is used transparently by msbuild when your Visual Studio projects are compiling. Looking inside the assembly's resource strings, we discover it:

Generates a .NET Licenses file and adds it to the manifest of the given assembly
Usage:
lc /target:TargetAssembly /complist:filename [/outdir:path] [/i:modules] [/v] [/nologo]

Options:
/target: Target assembly for the generated licenses file
/complist: Licensed component list file
/outdir: Output directory for the generated licenses file
/i: Specify modules to load
/v Verbose output
/nologo Suppress the display of the startup banner

The entry point into this assembly is the Main method of the System.Tools.LicenseCompiler class. Of (arguably) most importance is the /target: switch. This is the name of the assembly into which the compiled resource will be added. In Elsie.Target.exe this would be a resource named "Elsie.Target.exe.licenses", containing a binary stream representation of a serialized .NET object. More to come...

If you add a an empty text file named "licenses.licx" to your project, Visual Studio automatically sets its BuildAction:EmbeddedResource and CopyToOutput:DoNotCopy. It also calls lc.exe before calling csc.exe (yes, I'm a C#-a-holic). It makes the decision based on the .licx extension and you can have as many .licx files as you want in a single project (ok, that may not be true, but why would you want that many? Anyway, it will generate one /complist:[filename.licx] for each licx file in your project)

So what do you type in this/these text file(s)? If you really care, we'll have to make a 3rd installment.

Licensing Components in .NET - Part 1

I'm going to wear three hats here: the fantastic component developer who's written a third party library, and the poor sod who has to make the aforementioned fantastic component (not developer) work with the existing build environment. Finally, I'll be that guy who has to run this application and explain to his boss why it's suddenly stopped working.

Unsurprisingly, Microsoft do have a how-to for developing licensed components. There's also a nice diagram here. Some of the information presented in these tutorials is a little misleading, so I figured I'd get back to trying on my hats.

Hat 1

I've developed the one component that could change the world. That's a little c, not big C for component. There's no requirement for my class to fit into Microsoft's ComponentModel namespace (which is, incidentally, where all the bog standard licensing classes reside). So, I apply the [LicenseProvider(typeof(LicFileLicenseProvider))] to my fantastic class, and somewhere in the class I make a call to the static method LicenseManager.Validate(passing in the type of my class, as this is how the manager figures out that I wanted it to use the LicFileLicenseProvider). There are two overloads for Validate:
a) public static License Validate(Type type, object instance)
b) public static void Validate(Type type)
Option #1 offers the fullest functionality and it makes sense to make the call in my class's constructor - after all, the error message (of the LicenseException that's thrown if the call to Validate fails for some reason) WANTS me to do this: "An instance of type 'Elsie.Proprietary.Fantastic' was being created, and a valid license could not be granted for the type 'Elsie.Proprietary.Fantastic'. Please, contact the manufacturer of the component for more information." Because the call to Validate returns a new disposable License object, I'm responsible at least for ensuring it gets cleaned up properly. I'll assign it to an instance field, and make my class implement IDisposable.
Option #2 is a little less messy - I don't have to pass in an instance of my class, I don't have to worry about managing a Licence object's lifetime. "A valid license cannot be granted for the type Elsie.Proprietary.Fantastic. Contact the manufacturer of the component for more information."
That's it. I don't even have to create a license file.

Hat 2

I'm going to use the Fantastic class, so I mock up a new project of my own (which I call Elsie.Target.exe) and I add an assembly reference to it. Then I create (probably in notepad2.exe) a one line txt file: inside it I type "Elsie.Propietary.Fantastic is a licensed component". I make sure the file is called "Elsie.Propietary.Fantastic.lic" and I make sure it's copied to my working directory (probably by setting BuildAction:Content, and CopyToOutput:CopyAlways). Inside my application, I call the Fantastic constructor (within a using statement, because the class implements IDisposable, because the component deveoper was a responsible guy after all). Hidden inside the constructor, Fantastic checks if I'm allowed to use it by loading the .lic file. If the checks are successful, I go on my way to being a superstar developer. Otherwise, an exception will be thrown and it's back to the streets for me!

Hat 3

I'm in the London office at 7am. I deployed Elsie.Target.exe, along with Elsie.Propietary.dll and Elsie.Propietary.Fantastic.lic last night. While I've been sleeping, everyone in APAC has been delighted with just how fantastic the application is. In my excitement, I forget about being a cheeky monkey and changing the .lic file contents to read "... is *not* a licensed component". This is lucky for me, because it would BREAK the application!

Other examples:
Good: "Elsie.Proprietary.Fantastic is a licensed component."
Good: "Elsie.Proprietary.Fantastic is a licensed component. Yes it is!"
Good: "Elsie.Proprietary.Fantastic is a licensed component. NOT!"
Bad: "Elsie.Proprietary.Fantastic is a licensed component"
Bad: "Elsie.Proprietary.Fantastic is not a licensed component"
It turns out that the Key is valid if the text in the file starts with the fully qualified type name followed by " is a licensed component."

This is crazy! So crazy, in fact, that it might just work...

C# Verbatim Identifier

I've known for a long time that C# identifiers could have an optional @ sign in front of them, but until recently I thought that the character became part of the identifier.

int foo = 3;
int @foo = 4; //<-- error here: identifier 'foo' already in scope

So, it's really a way to call into code that may have been written in another CLR language that has identifiers that clash with C#'s reserved and keywords.

By way of example, to call the following VB.NET function

Public Class VerbatimIdentifierTest
Public Shared Function int(ByVal value As Double) As Integer
Return Convert.ToInt32(value)
End Function
End Class

from C# you'd invoke:

VerbatimIdentifierTest.@int(1.0);

Amazon EC2

Amazon's Elastic Cloud Compute (EC2) is a nice idea, but it's important not to overlook the "elastic" in the name. If there's an obvious temporal pattern to your service's usage (examples: 1. it could easily consume 100% CPU of 5 machines between 7 am and 11 am, but slow to a dawdle for the rest of the day, or 2. a large system could lie unused on weekends and holidays), and you design your application with parallelism in mind, this could be a cost effective strategy for hosting the service. As they bill you per "instance-hour" (loosely defined as an "instance" available for all - or part of - an hour), an available _idle_ instance will cost the same in a month as an available _fully-utilised_ instance. Costs start to fall as soon as you turn off your under-utilised instances dynamicall; which is something you can't yet do when renting physical space in a data center (AFAIK).

For small business without the need for elasticity (note: does that spell "unsuccessful"?), I'm not convinced it would be as cost effective as renting some tin in a data center and running multiple virtualized instances on some technology like Hyper-V, ESX or Xen.

Assert.That(actual, Is.EqualTo(expected).Within(tolerance))

New to NUnit 2.4 back in March 2007:

using NUnit.Framework.SyntaxHelpers;

I really like this idea as it can make some unit test code much more legible!

Language Optimization Observations

Progamming languages are becoming prolific. Most recently the trend of domain specific languages has emerged. On any given project, multiple languages will be employed (to varying degrees of success). Their design can be influenced by many factors. Some - if not most - languages enable one to succinctly and elegantly define a solution to a problem, speeding up initial development and subsequent modifications. However, as the languages get more abstract they appear to get slower and slower to run. Maybe it's that they're solving bigger problems? Maybe it's that their runtime environments have been designed for the general case, and literally "karpullions" of CPU cycles are being wasted maintaining state in which we [for a given application] are not interested. The smart money is on over-generalized and sometimes poorly implemented runtime environments.

Duct Typing

defn: A portmanteau of duck-typing and duct-taping. Used to describe the effects of defining .NET extension methods in your own code to give existing objects "new" functionality.

Closures in C#

If I'm using the wrong terminology: sorry; you lose. The C# compiler is a wonderful thing. Among it's many gifts to us are iterator blocks (think: yield return) and anonymous methods (think: delegate literals). Neither of these constructs has a direct parallel in MSIL, the common intermediate language to which all C# code is compiled. Instead the C# compiler defines new types to encapsulate their behaviours.

Iterator Blocks

An iterator block becomes a state machine; a disposable enumerator. Code defined in C# as this:

public static IEnumerable PublicEnumerable()
{
yield return 1;
yield return 2;
yield return 3;
}

is compiled down to the MSIL equivalent of this:

public static IEnumerable PublicEnumerable()
{
return new d__0(-2);
}

The class d__0 is a closure, and if we take a peek in Reflector we see the following definition:

[CompilerGenerated]
private sealed class d__0 : IEnumerable, IEnumerable, IEnumerator, IEnumerator, IDisposable
{
// Fields
private int <>1__state;
private int <>2__current;

// Methods
[DebuggerHidden]
public d__0(int <>1__state);
private bool MoveNext();
[DebuggerHidden]
IEnumerator IEnumerable.GetEnumerator();
[DebuggerHidden]
IEnumerator IEnumerable.GetEnumerator();
[DebuggerHidden]
void IEnumerator.Reset();
void IDisposable.Dispose();

// Properties
int IEnumerator.Current { [DebuggerHidden] get; }
object IEnumerator.Current { [DebuggerHidden] get; }
}

The object instance can be returned to the caller of PublicEnumerable where it will behave like the iterator block we know and love.

Anonymous Methods

Anonymous methods can access local variables defined in the same scope. When a delegate literal contains a reference to a method local variable (see: i in the following example):

public static void DelegateLiteralTest()
{
Int32 i = 5;
Action add = delegate(Int32 value) { i += value; };
Console.WriteLine(i);
add(1);
Console.WriteLine(i);
add(2);
Console.WriteLine(i);
}

the C# compiler generates a new type (closure) that resembles the following:

[CompilerGenerated]
private sealed class <>c__DisplayClass1
{
// Fields
public int i;

// Methods
public <>c__DisplayClass1();
public void b__0(int value);
}

Notice how the local variable is now scoped to the instance of the closure, and how it's an instance of this closure that's used inside the method:

public static void DelegateLiteralTest()
{
<>c__DisplayClass1 __closure = new <>c__DisplayClass1();
__closure.i = 5;
Console.WriteLine(__closure.i);
__cloure.b__0(1);
Console.WriteLine(__closure.i);
__closure.b__0(2);
Console.WriteLine(__closure.i);
}

I think this is pretty darn cool.

Singleton Pattern vs. C# Static Class

At the most basic level, both constructs are useful in limiting the number of instances of an object that can be created. There are numerous subtle differences at the implementation level, but the single biggest difference is at an object-oriented level: static classes do not support polymorphism. This stems from a combination of factors at the language level; in short, a static class cannot inherit from a base class (abstract or concrete) nor can it implement any interface methods. Conversely, you can neither define static virtual methods on a class, nor can you define them on an interface. So, if you're ever using dependency injection [insert link here] - a technique that relies heavily on polymorphism - you will likely find that static classes will be inadequate, limiting, and plain frustrating.

There are a number of other runtime level differences, but these are all secondary to the polymorphism issue. The simplest is that a static class can never be instantiated, and all methods are executed against the type instance. Contrast this with the singleton, of which exactly one instance is created, and where methods are executed against that instance.

Next time you're involved in object design and you come across a potential application for the singleton pattern, don't forget this posting!

Delegate vs. Event

What's the difference between a delegate and an event in C#, you ask? An event provides better encapsulation (read: a more object orientated approach) while a delegate is "merely" a immutable multicast type-safe function pointer. In other words, an event is an encapsulated delegate, where only the += and -= functions are non-private. That means that although we can add new subscribers to the event from another class, we cannot invoke it. The add and remove accessors are the only members to retain the event's declared access (e.g. public).

It's easier to explain delegates first.

Delegates

Function pointer: a delegate acts as a layer of indirection allowing references to methods to be passed between objects. The target methods are invoked when the delegate is invoked.

Type-safe: unlike function pointers in C++, a delegate knows the type of the target object, and the signature of the target method.

Multicast: a delegate maintains a list of methods; it isn't limited to invoking just one

Immutable: delegate instances - like strings - cannot be changed. When you add/remove another target method to/from a delegate instance, you're really creating a completely new delegate instance with a superset/subset of target methods.

A delegate example

Define the delegate type, or use one of the many predefined types shipped with the FCL.
public delegate void Function(string value);
Define the target methods that we can call. These will have to conform to the signature of the chosen delegate type.

static void WriteOut(string value)
{
Console.Out.WriteLine(value);
}

static void WriteError(object value)
{
Console.Error.WriteLine(value);
}

Define a class with a public delegate field (bad object-oriented encapsulation, but it's just for the purpose of example).

class DF
{
public Function function;
}

Construct a new instance of this class, add a couple of target methods and invoke them through the delegate:

DF df = new DF();
df.function += new Function(WriteOut);
df.function += new Function(WriteError);
df.function("Delegate");


Events

An event is a construct that is functionally similar to a delegate, but provides more encapsulation. Specifically, adding/removing target methods is given the accessibility of the event (e.g. public/protected/internal). Everything else is given the accessibility modifier of "private". This allows subscribers to be notified of an event, but not to trigger it themselves.

An event example

Again, pick a delegate type from the FCL or define your own. For this example, I will reuse the Function delegate type, and the two Function methods defined above.
Define a class with an Event:

class EF
{
public event Function function;
}

Construct a new instance of this class, add a couple of target methods:

EF ef = new EF();
ef.function += new Function(WriteOut);
ef.function += new Function(WriteError);

Because of the encapsulation, we cannot call the following method from outside the declaring type:

//ef.function("Event"); // ... can only appear on the left hand side of += or -= (unless used from within the event's declaring type)


A good technical resource:
Difference between a delegate and an event.

Indispensable Software Engineering Tools for .NET

Requirements
=
JIRA: rudimentary but ok, not so great for managing project plans
-
Source Control
=
Subversion: all check-ins must have a reference back to the original requirement.
Everything included in the release must be held in the versioned repository.
Everything means everything... environment specific config files.
(Don't store usernames and passwords in source control, but that's ok because you didn't hard code them into your configuration anyway, did you!?)
-
Build
=
CruiseControl.NET:
> get latest from source control, tag so that the build can be repeated.
> ensure compiled assemblies are versioned so they can be traced back to the build.
> build, run unit tests and other code metric tools.
Code Metric Tools:
>FxCop
>NUnit
>NCover
>anything for cyclomatic complexity?
Post Build
=
FishEye: great for seeing all metrics, committed changes etc. Useful when you need to see what changes went into a build.
-

Pulsed Threads

It happened quite a while back, but I thought I'd put a note up here as a reminder: a pulsed thread is still idle and another thread could acquire the lock. This blog entry is not about how best to re-write the producer/consumer problem as an event-driven model, it's to show how naive multi-threaded operations are - among other things - low hanging fruit when it comes to finding bugs...

Imagine a group of producer threads [P0..Pn] and a group of consumer threads [C0..Cn] writing to and reading from a queue of arbitrary size. Access to the queue needs to be thread safe, but the consumer threads could conceivably take a long time to process the items they consume from the queue. It is an error to attempt to read from an empty queue. For this example, we do not consider throttling the producers.

For a first (naive) attempt in C#:

01: // consumer
02: while (running)
03: {
04: Int64 value;
05: lock (queue)
06: {
07: if (queue.Count == 0)
08: {
09: Monitor.Wait(queue);
10: }
11: value = queue.Dequeue();
12: }
13: Console.WriteLine("{0}:{1}", threadId, value);
14: }

01: // producer
02: while (running)
03: {
04: lock (queue)
05: {
06: Int64 item = Interlocked.Increment(ref seed);
07: queue.Enqueue(item);
08: Console.WriteLine("{0}:{1}", threadId, item);
09: Monitor.Pulse(queue);
10: }
11: }

The result? After several thousand iterations we get a "Queue empty" exception on line 10 of the consumer. Puzzled we look a little closer: it's not immediately obvious. Of the 5 consumer threads (call it C1), one has acquired the lock, tested the size of the queue and put itself in a wait state until a producer thread (call it P1) has added an item. P1 pulses the queue to signal an item has been added. An extract from MSDN says:

When the thread that invoked Pulse releases the lock, the next thread in the ready queue (which is not necessarily the thread that was pulsed) acquires the lock.

So, while we were expecting C1 (waiting on the monitor) to acquire the lock, we were wrong. C0 was next in the queue for the lock.

To get the application running correctly, we'd need to change line 6 of the consumer to:

06: while (queue.Count == 0)

Kids, always read the label carefully!

Coupling: Interfaces and Abstract Classes

When "loosely coupled" is la mode du jour, nobody really likes an abstract class. Well nobody on my team. I suspect there's more going on here than the simple increase in coupling (e.g. the number of assumptions a caller must make) over calling an interface method.

Deriving from an abstract class:
1) forces you to inherit from an object hierarchy, and
2) forces you to provide an implementation for a set of abstract methods, and
3) usually means that some behaviour is "thrown in for free" in the base class(es).

Implementing an interface
1) forces you to provide an implementation for a set of method signatures.

So... use more interfaces; achieve a lower level of coupling, and you'll also implicitly be choosing composition over inheritance.

Exceptions

If only the System.Exception class were abstract or only contained protected constructors... This would force people to define their own custom exceptions (a good thing), but still wouldn't solve humankind's insatiable appetite for the catch-all exception handler.

System.OutOfMemoryException Part 2

We've been seeing loads of these recently, not as a result of memory leaks (the usual suspects) but the result of trying to do too much at one time. Now that 64-bit operating systems are here, couldn't we take advantage of the extra virtual address space? Microsoft have a comparison of 32-bit and 64-bit memory architecture for 64-bit editions of Windows XP and Windows Server 2003
outlining the changes between the two memory architectures. Next step is to see whether the 64-bit operating systems are approved.

UnhandledException

From .NET 2.0 onwards, unhandled exceptions on any thread cause the process to terminate. This is - according to a number of experts - the "correct" behaviour, and is different to how they were handled back in .NET 1.1.

The AppDomain.UnhandledException event allows you to perform some kind of diagnostics (such as logging the exception and stack trace, or performing a mini-dump) after such an exception is thrown, but the CLR is going to exit - whether you like it or not - just as soon as all event handlers have run.

So... if you are a responsible application developer who is spawning new threads (or queueing user work items to the thread pool) please please please ensure that _if_ you can handle the exception, that it is caught and not rethrown. Even if it means storing the exception and letting another thread handle it, as in the case of the asynchronous programming model (APM).

Goodbye to Awkward Dictionaries in .NET

I take enjoyment in analytic work, and often find myself writing code to empirically test a bunch of XML files to see if I've got the "implied schema" right. For example, I might have 500 files of similar structure, and I'd want to see the possible values (and count) of the "/Reports/Globals/@Client" node's value (using XPath). Usually this would entail using a Dictionary and every access would have to test if the dictionary already contains the key ... until now. After seeing that Ruby's designers allow you to specify a default value for a Hash, I took another step forward, allowing a user of the class to provide a default function, allowing more complex defaults such as empty lists.

using System;
using System.Collections.Generic;

namespace ConsoleApplication6
{
class Program
{
public delegate T DefaultFunction<T>();

public class RubyDictionary<TKey, TValue>
{
private IDictionary<TKey, TValue> inner = new Dictionary<TKey, TValue>();
private DefaultFunction<TValue> defaultFunction;

private static TValue Default()
{
return default(TValue);
}

public RubyDictionary()
{
defaultFunction = Default;
}

public RubyDictionary(TValue defaultValue)
{
defaultFunction = delegate()
{
return defaultValue;
};
}

public RubyDictionary(DefaultFunction<TValue> defaultFunction)
{
this.defaultFunction = defaultFunction;
}

public TValue this[TKey key]
{
get
{
if (!inner.ContainsKey(key))
{
inner.Add(key, defaultFunction());
}
return inner[key];
}
set
{
inner[key] = value;
}
}
}

static void Main(string[] args)
{
try
{
RubyDictionary<string, int> dict = new RubyDictionary<string, int>();
Console.WriteLine(dict["9"]);
dict["8"] += 1;
Console.WriteLine(dict["8"]);
dict["7"] = 5;
Console.WriteLine(dict["7"]);
dict["6"]++;
Console.WriteLine(dict["6"]);

dict["5"]--; dict["5"]--;
Console.WriteLine(dict["5"]);


RubyDictionary<string, IList<string>> lists = new RubyDictionary<string, IList<string>>(delegate() { return new List<string>(); });
lists["shopping"].Add("bread");
lists["shopping"].Add("milk");
lists["todo"].Remove("write this dictionary class");

foreach (string item in lists["shopping"])
{
Console.WriteLine(item);
}

RubyDictionary<int, int> foo = new RubyDictionary<int, int>(777);
Console.WriteLine(foo[8] - 111);
}
catch (Exception exception)
{
Console.Error.WriteLine(exception);
}
finally
{
if (System.Diagnostics.Debugger.IsAttached)
{
Console.WriteLine("Press enter to continue...");
Console.ReadLine();
}
}
}
}
}

IEnumerable vs IEnumerator

IEnumerator<T> generic interface defines "cursor" methods for iterating over a collection:
T Current { get; }
bool MoveNext();
void Reset();
void Dispose();

IEnumerable<T> generic interface defines just one method:
IEnumerator<T> GetEnumerator();

C# supports iterator blocks. These are blocks of code (i.e. method or property getter blocks) in which you "yield" a return value at various points in the code. The compiler uses hidden magic to create a nested type that implements both IEnumerable<T> and IEnumerator<T>; the body of your original method is stubbed out to call into this new object, and all your original logic is implemented in the IEnumerator<T>'s MoveNext method.

The C# foreach keyword is quite flexible and operates on any object that exposes a public GetEnumerator() method (the 'collection' pattern) - the object doesn't have to implement IEnumerable or IEnumerable<T>.

Experiment in the Fidelity of UTC DateTime

// make sure your local clock is not displaying GMT

DateTime a = new DateTime(2008, 7, 6, 4, 3, 2, DateTimeKind.Utc);
DateTime b = new DateTime(2008, 7, 6, 4, 3, 2, DateTimeKind.Local);
DateTime c = new DateTime(2008, 7, 6, 4, 3, 2, DateTimeKind.Unspecified);

// no surprises here...
Console.WriteLine(a);
Console.WriteLine(b);
Console.WriteLine(c);

// and again, no surprises. this makes it that much more explicit.
Console.WriteLine("-");
Console.WriteLine(a.ToUniversalTime());
Console.WriteLine(b.ToUniversalTime());
Console.WriteLine(c.ToUniversalTime());

// notice the Z(ulu) for UTC, the time zone for local, and the (nothing) for unspecified
Console.WriteLine("-");
XmlSerialize(a);
XmlSerialize(b);
XmlSerialize(c);

// ooh! our first loss of fidelity
Console.WriteLine("-");
DataSet dataSet = new DataSet();
DataTable dataTable = dataSet.Tables.Add();
DataColumn dataColumn = dataTable.Columns.Add("DateTime", typeof(DateTime));
dataTable.Rows.Add(a);
dataTable.Rows.Add(b);
dataTable.Rows.Add(c);
Console.WriteLine(dataSet.GetXml());

// those .net framework guys fight back with the DateTimeMode property (new in 2.0)
// understandably, weird stuff starts happening when not all input dates are of the same kind
Console.WriteLine("-");
DataSet dataSet2 = new DataSet();
DataTable dataTable2 = dataSet2.Tables.Add();
DataColumn dataColumn2 = dataTable2.Columns.Add("DateTime", typeof(DateTime));
dataColumn2.DateTimeMode = DataSetDateTime.Utc;
dataTable2.Rows.Add(a);
dataTable2.Rows.Add(b);
dataTable2.Rows.Add(c);
Console.WriteLine(dataSet2.GetXml());

//you can't just get the bytes for a DateTime, it's a struct.
//Console.WriteLine(BitConverter.ToString(BitConverter.GetBytes(a)));
//Console.WriteLine(BitConverter.ToString(BitConverter.GetBytes(b)));
//Console.WriteLine(BitConverter.ToString(BitConverter.GetBytes(c)));

Console.ReadKey();

Flex interaction with JavaScript (and ASP.NET)

It just doesn't quite work out of the box.

First, a flash object within an HTML form element fails when the flash runtime tries to register the callbacks you defined in your Flex app (e.g. a callback added in ActionScript using ExternalInterface.addCallback("someFunction", someFunctionPointer) is evaluated in the browser as __flash__addCallback(objectId, "someFunction"). A hack to get around this would be to assign your Flash object to window.objectId as soon as you've created it: hey presto! you've worked around one limitation.

Second. There appears to be a race condition if you need to initialize your Flex app with JavaScript data as soon as both are ready. At first, I thought I had a choice... push from JavaScript or pull from Flex. To push, we need to know that the callback has been added in the browser: either by polling or by registering an event. I refuse to poll, and I can't find a way to register for the event. The remaining option is to pull. The question is, will the source data (JavaScript) have been assigned by the time my Flex Application's applicationComplete event has fired? No idea, really.

Notes On the :: Operator in C#

Have you seen this construct before in auto-generated partial classes (e.g. code-behind .designer.cs for a web control)?

protected global::System.Web.UI.WebControls.TextBox MyTextBox;

Basically it's there to ensure that the System namespace is the same System namespace that's declared in the global namespace. So, why would they need this? Consider that you might define your new UserControl -derived class in a namespace like this:

namespace MyNamespace.System { ... }

Visual Studio's code-generator would create a partial class in the MyNamespace.System namespace, in which it would define a field MyTextBox of type System.Web.UI.WebControls.TextBox. However, when the time came for the compiler to put the puzzle together, the namespace resolution rules would match the "System" part of the member's type to the [unintended] "MyNamespace.System" namespace. Not cool.

Enter the global namespace alias, and the namespace alias qualifier operator. (Wow, that sentence is a mouthful).

To force the compiler to use the intended "System" namespace (which resides in the global namespace, not the MyNamespace namespace), the MyTextBox member is declared with the global:: namespace alias qualifier.

Obviously, things NOT to do include:
1) using global = System.Diagnostics;
Generates the compiler warning: Defining an alias named 'global' is ill-advised since 'global::' always references the global namespace and not an alias.
2) namespace System { class MyClass { } }
No warning from the compiler (Java doesn't let you do the equivalent).

Further usage patterns:

using sys = global::System; //<-- nice and recursive :-)
using diag = System.Diagnostics;
using dbg = System.Diagnostics.Debug;

sys::Object o = new System.Object();
diag::Debug.Assert(true); // <-- OK
dbg.Assert(true); // <-- OK, but important to make the distinction that it's a "type" alias and not a "namespace" alias.

More HTTP Compression in IIS 6.0

Further to my previous post, I've come across more issues with HTTP compression and IIS 6.0. IIS allows the compression of dynamic files (e.g. files generated on the fly by ASP.NET handlers), but Internet Explorer doesn't appear to like it when the dynamic file is a zip itself. Unfortunately, compression is controlled at the "Service" level (right-click on Web Sites, rather than right clicking on an individual web site) in IIS manager snap-in, and using shared infrastructure can mean that your web site breaks immediately for no apparent reason. Lucky for you though, the metabase can be tweaked to enable/disable compression at the "Site" level... you just have to be able to figure out the site number.

adsutil.vbs SET W3SVC/687245286/root/DoDynamicCompression False
adsutil.vbs SET W3SVC/687245286/root/DoStaticCompression True

IIS from the dark ages

Back in IIS 5.1 you can easily run multiple sites, just not all at once. Run the following scripts to enumerate the sites already installed, and copy an existing site to a new site!
[code]
C:\inetpub\AdminScripts>CScript.exe .\adsutil.vbs COPY W3SVC/3 W3SVC/4
C:\inetpub\AdminScripts>CScript.exe .\adsutil.vbs ENUM W3SVC /P
[code]

(De)bugger

A neat trick to assist debugging in C# or JavaScript, when you have no (or little) control over how or when the process is started: use the System.Diagnostics.Debugger.Attach() method or the debugger statement.

Visual Studio Dependencies

You're building a project named A.exe and you've included a file reference to the assembly B.dll (the value of the "CopyLocal" option is set to "true"). Assembly B references C.dll and D.dll, one of which is installed to the GAC. When you hit the build button, a copy of B is made in the project's output directory. Now comes the tricky part: without the ability to specify (or query) the "CopyLocal" option for the dependent assemblies, does msbuild.exe copy C.dll and D.dll? The answer: if the dependent assembly cannot be found in the GAC, then it's copied; otherwise it won't.

One more trick here: don't use Windows Explorer to try and view the GAC on another machine; you can't. Something about a shell extension? You can see the GAC with a command prompt and a mapped network drive.

To disable the shell extension, you can set the following registry value:
HKLM\Software\Microsoft\Fusion\DisableCacheViewer [DWORD] to 1. This will enable you to view the GAC of a remote machine.

Sharing the Strong Name Key File

Pedantry in motion: even though you shouldn't ever change your public key (see log4net versions 1.2.9 - 1.2.10 for an excellent reason) I still like to share a single strong name key file (.SNK) between all projects in a solution. Using Visual Studio 2005, this can be accomplished thusly:
1) Store the key in a parent folder of the project, so it can be accessed by all projects in the solution.
2) Add it as a link (use the add existing file menu)
3) In the signing tab of the project properties window, check the "sign the assembly" check box and choose the path from the drop down list. You will notice this is the path to the actual .snk file - not a local copy of that file.

Anonymous Methods

It looks like I've fallen off the bleeding edge again; this time I'm talking about anonymous methods: C#'s answer to the Java concept of anonymous inner classes. They've been available since C# 2.0 came out nearly two years ago. See this guy's blog for some cool stuff you can do with them. I've especially taken to sorting lists this way. :)

SQL Server 2005 Connections

So, you can see your local database server using SSMSEE, but when you fire up Visual Studio you can't find it anymore. First, you need to set up the surface area configuration to allow remote TCP/IP connections, then you need to configuration manager to choose a TCP port. If you choose 1433 then Hey, Presto! and everything should start working magically. If you intend to connect to the database server from another machine, don't forget to allow traffic over port 1433 using Windows Firewall.

Multi-stage deployments

I was recently foxed, I'll have to admit. Even when you don't think your own code's behaviour will impact another application, distributed shared environments seem to make it a likely event. For instance, you deploy a component upgrade onto one application server only... but you are unaware of the fact that references to your component are cached by clients, each of which can switch application servers at the drop of a hat. The result: thousands of errors an hour and irate support teams! :-)

Anyway, the lesson learned is that deployments - like code - should be thoroughly planned and all assumptions should be documented and passed over by someone who knows the target environment really well.

The other lesson learned is: don't change anything that you don't NEED to change. If you are required to alter the behaviour of one method, that's all you should do. Don't be clever and think that you can now refactor the code too; you'll get another opportunity to do that if it's really necessary. I guess it boils down to the fact that if your code was tightly coupled to start with - which, let's face it, a lot of code is - then you're more likely to break something. Something as silly as a client caching the method ID from an IDL definition instead of late binding using the method name can cause hours of anguish. Don't say I never warned you.

MSF Nostalgia

I'm determined not to make the same mistake again. A keen follower of process methodologies like RUP and MSF (and ACA.NET), I threw caution to the wind in developing a quote-unquote "prototype" recently. There wasn't much in the way of requirements, and it seemed like a cinch. Soon it became apparent that the business owners' visions weren't aligned and nobody could agree what the system should do. And I kept quiet because I had no idea what it should do; I'm the technology guy who would put it together from a requirements document. WRONG! What I should have done is forced them to agree on the vision ... then on the behaviour ... then on the look and feel. Lesson #1: to have their sign-off on these documents would have made my life a lot easier in the final stages where we quibbled over the system's behaviour (my interpretation of their ever-changing minds). Lesson #1.5: to get the sign-off of more senior people than those I dealt with every day would have more impact than the easily vetoed sign-off of not-so-high-up-management. Lesson #2: most importantly, have a signed-off requirements document. This you can use as a test plan, and you can use to check the boxes that everything you promised has been delivered. In summary, I think I delivered far too much functionality for free - but that's not necessarily good for the business because there was never a 100% clear and auditable understanding between myself and them. Lesson #3: never underestimate the importance of "I'd like that in writing".

IExecutionStep

Why write a blog entry about the ASP.NET HTTP pipeline when someone else has already done it better? I'm talking about Rick Strahl's Low level look at ASP.NET architecture. Note to self: as soon as you have a box running IIS 7, see if you can't reverse-engineer everything down so far as to understand even the IExecutionStep interface.

Hidden Face of NTFS Permission Issues

How can everyone be so naïve about NTFS permissions? Just yesterday I had an Associate Director bring a web site to its knees by cut-and-pasting a configuration file from somewhere else on the system. Today's problem was far more obfuscated though, and we would have saved six hours of 3 devs' (and countless users') time if the same (recently promoted) Associate Director had looked in the event log as suggested to him. As it turns out, when the IUSR_MACHINENAME user account is locked out - and IIS is configured to only allow anonymous access - IIS will not serve any static content. If you allow Integrated Windows Authentication as well, then IIS will fall back to that mechanism (a corollary: if your service delivery guy is an administrator, everything will appear fine to him). In the first instance, a Filemon trace won't show failures of w3wp.exe trying to access the files, because it's the logon that's failing - and this can be very confusing! In the second instance, it will clearly display the NT username (of the token that w3wp.exe's associated with the current request) next to the file open failure.

So, next time you see the error: access denied, and you've ruled out the fact that a Muppet has messed up the permissions on a critical file... look in the Event Viewer and save yourself a bunch of time. It's there for a reason. Use it. Take a step further even, and write to it yourself!

HTML, JavaScript and Background-Color

Well this had me confused for an hour or so. Consider the HTML element below:

<div style="background-color:red">

First, I tried to set the color programmatically using JavaScript. I knew that layer.style.background-color wouldn't work so I typed in layer.style["background-color"] = "green"; No luck. Eventually, I noticed that layer.style["background"] worked, and I went on my merry way. For about 30 seconds. Now I needed to read the background-color value from a layer, and my previous two attempts were turning up empty strings.

With a bit of Googling, I discovered that background-color should be accessed as layer.style.backgroundColor (or layer.style["backgroundColor"]).

Google saves the day.

(A couple of weeks later, and programmatically setting the css class of an HTML element has tripped me up. It seems the property is called "className" in JavaScript.)

???

It's pretty irritating to have someone else's XML document object model - on a remote server with little in the way of a debugging tool kit - screw up your document by replacing all non-ASCII characters with the ? (Question Mark) character. It's especially annoying if your application is used globally by important clients and their data contains lots of accented characters. But it's better to spit out loads of question marks than to completely F&*^ things up by just chewing off the most significant bit each time a non-ASCII character comes along (ASCII characters are all encoded using the 7 least significant bits of each byte). The first issue is a display problem that will usually only be caught by the human eye. The second one affects other law-abiding XML parsers because they are being fed invalid XML documents. A client is more likely to forgive a couple of question marks on the screen - at least they can see the rest of the data - than they are to forgive an error page because their document couldn't even be parsed.

CallContext

Today, I discovered CallContext, and it is good. I will probably use it lots instead of being tied to HttpContext. Sure, there's a downside in that we won't be able to switch threads freely, but hey, at least it works in non-Web environments, so I don't have to reference System.Web.dll from all my core and application logic classes.

2 days later: I have torn out most of the remaining hairs from my scalp. It turns out that ASP.NET doesn't - in fact - guarantee that your entire request will be executed on the same thread (see here for more details). At least the HttpContext class still works; even though it depends on an underlying CallContext (when an HttpApplicationFactory initialises an HttpApplication, the new HttpContext ...blahblahblah I'm tired) the HttpContext is moved from thread to thread whenever one of this switches occurs.

What I have learned: don't trust simple Google searches that tell you each ASP.NET request will only run on a single thread.

System.OutOfMemoryException

So you've run out of memory. The question is: what kind of memory don't you have enough of? The answer - it seems - isn't "physical", it's actually "virtual". Most 32-bit Windows processes are limited to 2GB of user-mode virtual address space (you can increase this to 3GB if you know what you're doing), and this is what you've just exhausted. If the CLR cannot find a contiguous section of free virtual memory to allocate for a new object - BANG! - the exception gets thrown.

Update: http://msdn.microsoft.com/msdnmag/issues/06/11/CLRInsideOut/default.aspx describes a second reason: "or there is not enough physical memory available in order to commit."

NUnit and Visual Studio 2003

In any typical application that I'm asked to develop, there is an assembly of business objects I'm compelled to unit test. So I create a new project to the solution with ".Test" appended to the name (e.g to unit test the CDMA.Business assembly, I'd create a new CDMA.Business.Test class library project that references both the CDMA.Business and nunit.framework assemblies). Frequently, the subject of unit tests depends on application configuration settings, so I'll create a new file called CDMA.Business.Test.dll.config (note that this is the name of the assembly, with ".config" appended) within the test project and then I'll add a new Post-build event to copy the file to the target directory. Project Properties -> Common Properties -> Build Events -> Post-build Event Command Line:

xcopy /y "$(ProjectDir)$(TargetFileName).config" "$(TargetDir)"

To enable debugging via the F5 key, I'll use the following settings in Project Properties -> Configuration Properties -> Debugging:

Debug Mode: Program
Start Application: C:\Program Files\NUnit 2.2.8\bin\nunit-gui.exe
Command Line Arguments: CDMA.Business.Test.dll

If you then set your unit test project as the start up project you can just press the F5 key and Visual Studio will automagically build the project, copy the configuration file, run the nunit-gui executable and attach the debugger to it. To skip the last step, just press CTRL-F5 instead.

C# 3.0

Ted Neward explains how the new features are nothing more than syntactic sugar that makes programmers' lives a little easier. For example, all the IL code produced by the C# 3.0 compiler can be reverse engineered by Lutz Roeder's .NET Reflector to show you how the feature's actually been implemented. Brilliant.

Extension Methods to the Rescue

Today I added some caching to a prototype application I'm developing at work. Microsoft Enterprise Library was my implementation library of choice, but I was frustrated by the choice of available overloads for CacheManager.Add(). Rather than add my own overload, spoil the clean interface and recompile the assembly, I ended up writing a single method on a static class that took an instance of the CacheManager, key, value and expiration as parameters, adding in the default values in a call to the CacheManager object. Now this may not be object-oriented, but it's very similar to how extension methods work in C# 3.0 (in my opinion, they exist solely to _fake_ the object-orientedness in calling code). In fact, when you declare a class like this:

public static class CacheManagerExtension
{
public static readonly TimeSpan DefaultAbsoluteExpiration = TimeSpan.FromMinutes(5);
public static void Add(this CacheManager cacheManager, string key, object value, TimeSpan absoluteExpiration)
{
cacheManager.Add(key, value, ..., new AbsoluteExpiration(absoluteExpiration), ...);
}
}

the new C# 3.0 compiler (figuratively speaking) recognises the "this" keyword in the parameter list, and marks the method, class and assembly with the ExtensionAttribute custom attribute. Now, you can call the Add method like this:

cacheManager.Add(key, value, CacheManagerExtension.DefaultAbsoluteExpiration);

and the compiler (and hopefully Intellisense) - upon realising that no method with that signature exists on the CacheManager class - will translate it into a call on the static class marked with the ExtensionAttribute attribute.

Oh yeah, and my other gripe with Microsoft Enterprise Library is the fact it's caching is not aspect-oriented, but that's the topic of another post...

Post-script: Don't blindly assume that your service delivery team are happy to install EntLib on the production server. :-)

Using log4net

log4net is my recommended lightweight tool for emitting log statements from .NET code. I'm a firm believer that application users shouldn't see exception messages or stack traces - that information is for system administrators and developers only. I like storing log information in a database or in a text file on the server and allowing developers to view it when things start to go pear shaped.

Here's how I typically set up log4net on a project:

- Create a sub directory within your solution - called 3rdParty - and copy the log4net assembly there. This allows meto store one copy of the assembly, and reference it from all projects in the solution.
- Create a separate configuration file that I call log4net.config (I don't use Web.config or App.config)
- Use the XmlConfigurator custom attribute on the assembly that will be loaded first
(Don't forget to set the Watch property to true)
- Start appreciating the benefits of logging!

NOTE 1: If you do choose to use Web.config or App.config, the only reason you have to declare the IgnoreSectionHandler is for .NET to ignore the log4net section!

NOTE 2: No, I don't know what line of code causes the XmlConfiguratorAttribute to be reflected and subsequently configured.

C# 3.0 is not dotnetfx3; repeat;

I am a little behind the times, it seems, as Microsoft released .NET Framework 3.0 just a few days ago. Luckily, the release is true to it's name - it's just another version of the framework with the recent additions WCF, WWF, WPF and CardSpace(collectively and formerly known as WinFX). It contains neither C# 3.0 nor LINQ, both of which will be released with "Orcas". Phew. That means I have time to brush up on my knowledge of automatic type inference before a random client starts asking tough questions!

Technical Forum

If I could offer you only one tip for the future, ______ would be it. I've tentatively accepted a spot to speak on a technical subject at a forum of consultants, but I haven't yet chosen the topic. It's my first foray into the world of presentations; I don't expect it to go swimmingly but if I can judge the level of my audience and give them something useful then my job will have been done. Actually, I will judge it a success if the audience can go away and apply some practise they learned from me (or were inspired to learn by the topic).

To meet this goal, the focus needs to be on both the content and its delivery. I hope I can do it!

Exception Handling

OMF! My blood reached boiling point this week when I overheard a colleague whining when his poorly structured error handling and logging mish-mash was removed from the codebase. Similarly, it boiled a couple of months ago when a group of fellow consultants were encouraging a user interface element that would generically display stack trace information to users. Please, people, follow these guidelines:
a) Stack trace (or even exception type information) should NEVER be shown to users because you don't know what kind of sensitive (perhaps proprietary) information is contained within.
b) DON'T append long dynamic SQL statements (or any other information used solely for diagnostic purposes) to your Exception's Message property so that it can be displayed on the screen, unencrypted, rather log it separately.
c) DO use a framework like Microsoft's Enterprise Library, or a component like log4net to trace information rather than displaying it in the end user's UI.

Inertia Xenon

I released the first alpha version of Inertia Xenon - my lightweight .NET grid framework - this afternoon. Now that it's out, I need to focus my attention on the parts that require it the most. Initially, I feel that the proxy between the client and service broker could be made a lot better if I baked in a whole lot of multi-threading goodness. Next, I bet my job will be focused on improving the administrator API. Anyway, that's all just speculation. Try it out today and let me know what you think..

Defining Architecture

I'm recently attended one of a series of sessions aimed at developers who are aspiring technical architects. From previous experience, I'd decided that the IEEE definition of architecture (presented below) is spot on; now I just wish that others would subscribe to the same idea.

Architecture is the fundamental organization of a system embodied in its components, their relationships to each other, and to the environment, and the principles guiding its design and evolution. [IEEE 1471]

It bothers me when junior developers use the word to buff up their resumes. It bothers me more when experienced developers are called Technical Architects by their consultancy; they should know better. Architects, by my definition, are those people who specialize in architecture as a profession. They are not the people who can recite a few patterns and have skimmed through a couple of Martin Fowler's books (although both of these traits might be present in a real architect). Hmmm. Let's see if the next session is any better.

Services Share Schema and Contract Not Implementations

It hit me like a 10 ton truck, while trying to architect an grid computing framework for Microsoft .NET that will be service-oriented. It is wrong for a client of a service to depend on any behaviour of its own implementation of, for argument’s sake, an input parameter. The initial grid’s design saw heavy usage of the Command pattern to pass not only data but behaviour across the service boundary. It all falls apart however as soon as you have to make the service boundary more explicit (i.e. J2EE servicing .NET clients) – at that point I realized I needed to revisit the four rules for service-orientation. I now know why wsdl.exe generates you a new set of data transfer objects ... and why SOAP defines request and response messages the way it does. And it's all becoming very clear to me, and funnily enough typing this blog reminded me that this was an interview question I was asked over a year ago by Josh! I’m going to stick the four tenets of service orientation on Post-It notes to my monitors.

Boundaries are explicit
Services are autonomous
Services share schema and contract, not class
Service compatibility is based upon policy

Other Side of the Fence

Phew! I've passed the Java certifications in JSP, Servlets and EJB and can now return to the normal way of life using Microsoft technologies to my heart's content. I'm keen to get back to IIS although I wouldn't mind working with some of the J2EE application servers to see value-add features that each vendor has been able to provide (all my exposure so far has been to Sun's bare bones reference implementation). Expect more Microsoft-related posts soon!

Throttling Paradox

Who would have thought that you might need to throttle an application server in order to maintain a performance-level contract? I've seen the option inside IIS but couldn't for the life of me think why anyone would want to turn it on. However, in the context of an application that's been through some sort of capacity planning process (so we have a rough idea of how many requests/responses the application can process), and where all requests need their responses within some limited duration (think high-volume/low-latency trading platforms), you can apply throttling to help enforce the performance level. Without throttling, additional requests might be satisfied only to the detriment of other requests already executing on the application server. Ideally, it should be used in conjunction with load-balancing across a cluster of servers, so that throttled requests aren't queued, but rather dispatched to another less-busy machine.

Rfc2898DeriveBytes

Gone are the days of salting and hashing users' passwords and storing the salt and salted hash next to each other in a database. .NET 2.0's new Rfc2898DeriveBytes class derives a pseudo-random key from a password, salt and a number of iterations - a so-called iterated and salted hash. And, apparently, it's more secure than just hashing a password and salt. More info here.

CAFEBABE

I just noticed while studying for tomorrow's exam that every Java class file begins with the so-called magic number 0xCAFEBABE! Quite funny, especially since the guys who penned it into the class file specification did so before the language was even called Java. You can find more information at Artima.

.NET 2.0 vs. Java 5

Type erasure, reification and synthetic bridge methods: yuck. I'm stuck on the proverbial consultants' bench so I decided - after 4 years of non-Java development - to upgrade my Java certification to the latest version in just five days. I honestly thought I was going to make the deadline until I hit the section on generics. Now I'm a half-day behind and my head hurts! C# is way easier to grasp, and from the looks of things performs a whole bunch better too (except for the JITed type instance "explosion" if you really care).

Anyway, since Java 5, the Class class in package java.lang is a generic class whose type parameter denotes the type that the Class object represents. Previously, compilers wouldn't let you compare an Integer and a String. Now (since Tiger) they won't even let you compare Integer.class and String.class! It stems back to the fact that (similar to C#) both GenericType<A> and GenericType<B> do not derive from each other, even though type B extends type A. There's a little bit of extra trickery in the Java though, because both GenericType<A> and GenericType<A> extend GenericType<?> (unlike C# where they derive from System.Object.) This means that you CAN compare a GenericType<capture of ? extends Object> with a GenericType<capture of ? extends String>! Madness.