Saturday, February 28, 2009

LiveCycle Data Services / amd64 / ubuntu

I haven't tried out the alternative - BlazeDS - yet, but I've had the time of my life trying to get ubuntu to use all my RAM and both my monitors, and even more fun trying to get LCDS installed. Nothing has been exceptionally difficult, but I've noticed a lot of things just are not compatible. Knowing not only where you're going, but also the route to get there, can make a major difference in total time taken. That's why I document my "mistakes" - for next time. Next time...

  1. Well, maybe next time I'll get Hyper-V Server 2008 to work. Even VMWare ESXi wasn't happy with my SCSI drives. Xen looked like more than I was prepared to bite off, nevermind chew.

  2. Accessing any more than 3GB of physical RAM on an ubuntu install requires the amd64 package. Similar limits are imposed on other 32 bit operating systems. In this case I chose to upgrade to the 64 bit version than enable PAE.

  3. to do anything useful in an unfamiliar operating system you'll want to use the desktop version.

  4. the graphics driver and operating system seemed to fight like small children if the driver's not open source ... I'm just glad that nVidia have made the effort for a 64 bit driver so I can see my desktop in TwinView, rather than being forced to run the server version (see 3).

  5. the Flash plugin that you can download from Adobe's website is - at time of writing - only compatible with i386 and not amd64. Problem's easily solved by using sudo apt-get install flashplugin-nonfree which was already set up inside Aptitude.

  6. when you download a .bin file, you're expected to know you're going to chmod +x and execute it. However, in the case of lcds261-lin.bin you can't just do this inside a terminal from the CTRL-ALT-F7 session; you must run it from one of the CTRL-ALT-F{2..6} sessions. Else you'll get a Java error java.lang.UnsatisfiedLinkError: Can't load library: /usr/lib/jvm/java-6-openjdk/jre/lib/amd64/xawt/libmawt.so. Amusingly, one of the lines in the stack trace read: at ZeroGay.a(DashoA10*..)

  7. even if you asked the installer for a Tomcat install, don't expect your container to be started when installation is complete. Perhaps it's too much to expect a reasonable error message too. Anyway, I created a new service account user "tomcat" with sudo adduser tomcat and then gave it ownership of the (otherwise locked down) Tomcat install directory and started the server.

  8. Actually check that the samples are working - front-to-back - at this point. First time around, I celebrated an early victory when I first saw the sample pages running inside Tomcat. There was a leeeettle bit more setup to do re: the sample database. The LCDS install ships with an HSQLDB database, which needs write permission on a bunch of directories. In the spirit of the previous step (and, perhaps, a pre-existing condition in which I derive pleasure from creating new service accounts) I created a new user called hsqldb with permission to create the .lck (lock) files. sudo adduser hsqldb and then /opt/lcds/sampledb$ sudo chown -R hsqldb .

  9. LCDS is free under Adobe's developer license on more than 1 CPU (or even in production environments, but only on a single CPU). This is great for developers like me who can tinker at home without parting with hard cash. We can even demo our applications to clients for free. The people with the cheque books can keep them in their pockets until bigger test and production environments are commissioned.

Friday, February 27, 2009

System.OutOfMemoryException Part 3

It's not every day you come home and plug 8GB of physical memory into your home PC. Before now, I wasn't completely aware that there was even a limit to the amount of memory you could usefully install on a Windows XP machine. There are all sorts of hurdles it seems. Luckily, Windows Server 2008 x64 has no problem with addressing it all. Bonus. But can you imagine the size of the swap files you would start to rack up if you ran a bunch of memory hungry processes simultaneously. Processes only see virtual memory... Windows is smart about whether this memory is backed by physical memory pages, or pages allocated on a system file called the swap file. As we move to the wider 64-bit addresses, we're not going to get another System.OutOfMemoryException; instead we'll run out of disk space for the swap file or spend so much time thrashing that we'll want the small address space back!

Saturday, February 21, 2009

Gen / Spec

I think it's common for vendors and consultancies to push their own technologies as solutions to the problems of clients. Even individual consultants are often bullish about the skills they've acquired. Together, these behaviours make it difficult (if not impossible) for the optimum solution to be found and implemented. Of course, an optimum solution isn't always what a client wants: consider delivering a sub-optimal solution that leaves your client better off than the nearest competitor. Still, I feel that recognising that an optimal solution does exist is an important step towards better architectures.

Adam Smith - the economist - wrote about the division of labour in his magnum opus, The Wealth of Nations. To paraphrase: he talks of a pin factory and the difference in productivity between two scenarios - sharing responsibilities across a group of employees, and assigning specific tasks to employees. There are many conclusions one can draw, but the one that shines out particularly to me is that performance gains can be made by choosing the right tool for the job (RTFJ).

Databases. Great for relational data storage and retrieval. They're designed to meet ACID requirements, and even with all the log shipping and replication in the world, don't scale quite as nicely as other technologies can/do. In my book, that would have been reason enough to keep business logic out of the database. However, certain cases might call for a gargantuan set of data to be worked on at once, and it might be prudent to "bring the computation to the data".

Grids and high performance computing. Great for compute intensive operations. However they're distributed by nature, and that generally makes things more difficult. They usually offer only a subset of the common operating system constructs we're used to - well conceptually, anyway. Spinning up a new thread locally is the "equivalent" of starting up a new process on a remote machine. Also there's the problem of moving data. (De)serialization is computationally intensive - optimisations can take the form of using shared metadata of common versions (e.g. .NET assemblies, and binary serialization) which bring new problems of managing versioning across the environment.

Whatever you're doing, always make "efficient" use of your CPU. Use asynchronous patterns for non-CPU tasks (e.g. waiting on I/O) using callbacks. Thread.Sleep() and spinning in a tight loop are generally evil (but I'm sure there exists a case where both are specifically wonderful).

Distribute only what you have to. If your constraint is virtual ("addressable") memory then it might be OK just to have multiple processes on the same machine with lots of physical memory, talking to each other via some non-network IPC mechanism.

Cache hits should be quick. Cache misses (generally) shouldn't result in multiple simultaneous requests to insert fresh data in the cache. Tricky bit is not making any "threads" wait while the cache data is inserted. That ties my previous point in with the next:

DRY. Don't repeat yourself. This goes for operations as well as for boilerplate copy and pasted code. If a cache can give you the result of an expensive operation you've already computed, for less cost, then consider caching. In-memory, disk, distributed and bespoke caches exist. Each will have a

Thursday, February 12, 2009

Licensing Components in .NET - Part 2

I reckon the only reason LicFileLicenseProvider is part of the framework is to get the point across the licensing greenhorns. All of a sudden, brains start ticking: you can load the license from anywhere. You begin crafting nefarious schemes using public key cryptography. It's brilliantly academic. But something's wrong. It would be easier just to hack the code. Steal the intellectual property. Hey, maybe that's why Microsoft stopped at the LicFileLicenseProvider? Maybe, and here's a thought: maybe they should have.

Another crazy piece of the (even crazier) licensing puzzle: licenses.licx files and lc.exe.

lc.exe is a tool written in .NET by Microsoft, which is used transparently by msbuild when your Visual Studio projects are compiling. Looking inside the assembly's resource strings, we discover it:

Generates a .NET Licenses file and adds it to the manifest of the given assembly
Usage:
lc /target:TargetAssembly /complist:filename [/outdir:path] [/i:modules] [/v] [/nologo]

Options:
/target: Target assembly for the generated licenses file
/complist: Licensed component list file
/outdir: Output directory for the generated licenses file
/i: Specify modules to load
/v Verbose output
/nologo Suppress the display of the startup banner

The entry point into this assembly is the Main method of the System.Tools.LicenseCompiler class. Of (arguably) most importance is the /target: switch. This is the name of the assembly into which the compiled resource will be added. In Elsie.Target.exe this would be a resource named "Elsie.Target.exe.licenses", containing a binary stream representation of a serialized .NET object. More to come...

If you add a an empty text file named "licenses.licx" to your project, Visual Studio automatically sets its BuildAction:EmbeddedResource and CopyToOutput:DoNotCopy. It also calls lc.exe before calling csc.exe (yes, I'm a C#-a-holic). It makes the decision based on the .licx extension and you can have as many .licx files as you want in a single project (ok, that may not be true, but why would you want that many? Anyway, it will generate one /complist:[filename.licx] for each licx file in your project)

So what do you type in this/these text file(s)? If you really care, we'll have to make a 3rd installment.

Licensing Components in .NET - Part 1

I'm going to wear three hats here: the fantastic component developer who's written a third party library, and the poor sod who has to make the aforementioned fantastic component (not developer) work with the existing build environment. Finally, I'll be that guy who has to run this application and explain to his boss why it's suddenly stopped working.

Unsurprisingly, Microsoft do have a how-to for developing licensed components. There's also a nice diagram here. Some of the information presented in these tutorials is a little misleading, so I figured I'd get back to trying on my hats.

Hat 1

I've developed the one component that could change the world. That's a little c, not big C for component. There's no requirement for my class to fit into Microsoft's ComponentModel namespace (which is, incidentally, where all the bog standard licensing classes reside). So, I apply the [LicenseProvider(typeof(LicFileLicenseProvider))] to my fantastic class, and somewhere in the class I make a call to the static method LicenseManager.Validate(passing in the type of my class, as this is how the manager figures out that I wanted it to use the LicFileLicenseProvider). There are two overloads for Validate:
a) public static License Validate(Type type, object instance)
b) public static void Validate(Type type)
Option #1 offers the fullest functionality and it makes sense to make the call in my class's constructor - after all, the error message (of the LicenseException that's thrown if the call to Validate fails for some reason) WANTS me to do this: "An instance of type 'Elsie.Proprietary.Fantastic' was being created, and a valid license could not be granted for the type 'Elsie.Proprietary.Fantastic'. Please, contact the manufacturer of the component for more information." Because the call to Validate returns a new disposable License object, I'm responsible at least for ensuring it gets cleaned up properly. I'll assign it to an instance field, and make my class implement IDisposable.
Option #2 is a little less messy - I don't have to pass in an instance of my class, I don't have to worry about managing a Licence object's lifetime. "A valid license cannot be granted for the type Elsie.Proprietary.Fantastic. Contact the manufacturer of the component for more information."
That's it. I don't even have to create a license file.

Hat 2

I'm going to use the Fantastic class, so I mock up a new project of my own (which I call Elsie.Target.exe) and I add an assembly reference to it. Then I create (probably in notepad2.exe) a one line txt file: inside it I type "Elsie.Propietary.Fantastic is a licensed component". I make sure the file is called "Elsie.Propietary.Fantastic.lic" and I make sure it's copied to my working directory (probably by setting BuildAction:Content, and CopyToOutput:CopyAlways). Inside my application, I call the Fantastic constructor (within a using statement, because the class implements IDisposable, because the component deveoper was a responsible guy after all). Hidden inside the constructor, Fantastic checks if I'm allowed to use it by loading the .lic file. If the checks are successful, I go on my way to being a superstar developer. Otherwise, an exception will be thrown and it's back to the streets for me!

Hat 3

I'm in the London office at 7am. I deployed Elsie.Target.exe, along with Elsie.Propietary.dll and Elsie.Propietary.Fantastic.lic last night. While I've been sleeping, everyone in APAC has been delighted with just how fantastic the application is. In my excitement, I forget about being a cheeky monkey and changing the .lic file contents to read "... is *not* a licensed component". This is lucky for me, because it would BREAK the application!

Other examples:
Good: "Elsie.Proprietary.Fantastic is a licensed component."
Good: "Elsie.Proprietary.Fantastic is a licensed component. Yes it is!"
Good: "Elsie.Proprietary.Fantastic is a licensed component. NOT!"
Bad: "Elsie.Proprietary.Fantastic is a licensed component"
Bad: "Elsie.Proprietary.Fantastic is not a licensed component"
It turns out that the Key is valid if the text in the file starts with the fully qualified type name followed by " is a licensed component."

This is crazy! So crazy, in fact, that it might just work...