Dare Obasanjo compares database caching with how compilers manage the various CPU caches (e.g., L1, L2). Surprisingly he comes to the conclusion that you need to implement your own caching scheme through memcached and friends because in the database situation the amount of data is so large:

The biggest problem is hardware limitations. A database server will typically have twenty to fifty times more hard drive storage capacity than it has memory. Optimistically this means a database server can cache about 5% to 10% of its entire data in memory before having to go to disk. [...] So the problem isn't a lack of transparent caching functionality in relational databases today. The problem is the significant differences in the storage and I/O capacity of memory versus disk in situations where a large percentage of the data set needs to be retrieved regularly.

I wonder how this is different from the sizes of L2 cache compared to main memory?

It's even worse: on a regular PC you might have 4MB L2 cache, but 4 GB of main memory. That's about 0.1 % compared to main memory - so databases actually have a relatively luxurious position, compared only by relative data size.

Application knowledge

Quite the contrary, I believe the problem is not in the data sizes, but in the optimization hints available to databases (and potentially the smartness of the database caching methods). With a good compiler, in particular a JIT, you can easily judge what data will be used in the near future in the code execution, and through data flow analysis and fancy register allocation tricks a compiler has a pretty complete knowledge of what the code tries to do, so it can optimize cache usage (and a whole lot of other things) very efficiently.

Compared to this, databases have little knowledge of the data access patterns of the application. They can only rebuild this knowledge from observations on the queries hitting them, but they don't seem to be very successful there judging from the frequency you hear people talking about memcached. I'm not sure why that is exactly, maybe because it's always more difficult to implement optimizations based on observations of dynamic data than on static knowledge?

One possible problem is probably that the database doesn't necessarily know - and in many situations probably cannot even guess - which data structures are commonly displayed together for a certain webpage, so caches can be invalidated together, or directly stored together.

It might be an interesting thought experiment to think what would be possible if the database was integrated with the application logic in a way that would make the application knowledge available to the database. This could probably lead to interesting changes regarding invalidation and cache organization. I know there were (and probably still are) some things going into this direction in the Smalltalk environment, but I have no idea if they really take advantage of the application knowledge. Probably not, as most Smalltalk is highly dynamic and I don't think they've put the emphasis into declarative programming that would be needed for this.

Cache Granularity

Also interesting might be the fact that the granularity of objects in a relational database is quite different from the application perspective. Relational databases store entities in rows in tables, but (web-)applications have a hugely different data model. A single application entity, e.g., a user, will span over several tables. But the pages the database uses as the unit of caching usually only contain data from one table. If you have 16k of cache, that might be enough memory for several hot data entities, but because the database caches tables, not application entities, much of those 16k will be filled with rarely used rows, e.g., 4k from the users table, 4k from the mood messages table, 4k from the friends table, and so on. Application developers fight this (and the processing cost of joins operations) with denormalization, which is basically a hack to reduce the number of tables an entity spans.

This all boils down to the fact that relational databases were designed for mass-data processing, like in financial institutions, where large calculations over huge tables of uniform data with little nested structures are the common operation.

I think this is one of the area where non-relational databases, like XML databases, are going to have a bright future. The data model, and thus the unit of caching, is much closer to what today's content-centric application's data actually looks like. It's not only much easier to program without that impedance mismatch, it can also have significant performance advantages over RDBMSes.

Some of you already know it: I've joined EMC Corporation, starting on Oct 15th. I'll be teleworking from Potsdam but commuting every month to Rotterdam for one week, which is an ideal arrangement for me.

EMC acquired X-Hive, my former employer in the Netherlands, in summer last year, so I'm basically returning to the company I left some time ago, mostly to finish my studies and try some stuff out (like taking a look into SAPs corporate innards, doing some freelance work).

I'm very happy that this worked out. X-Hive used to be a great employer, with really nice people and very interesting work. And now they suddenly pay much better ;-). Seriously: from my first days, it seems as if the influence of EMC is very good. Of course there is some corporate beaurocracy (which so far seems very acceptable), but on the other hand there are some seriously smart people giving input into my favorite native XML database. There is a huge set of really cool requirements to match, and X-Hive/EMC now certainly has the resources to fulfill them.

I'm very happy to work on this really cool product once again, and I'm particularly happy that it is probably going to have a much larger impact very soon. Nice times.

I'm happy to report that I tested Time Machines Backup-Restore capability yesterday evening, and it works, sigh.

I brought my MacBook Pro (1st gen.) in for repairs, because the left fan failed again, and I also had them build in a bigger hard drive. Restore from Time Machine took about 2.5 hours, maybe a bit more, but afterwards you're directly booting into your complete system. Very nice!

On the fan: Apple decided to fix it on guarantee, as the very same fan failed a bit less than two years ago. Also, not long before that fan, the right fan failed. So Apple, you're building a computer that sells for more than 2000 €, and you cannot build/buy/ship fans that last more than a year?

Update: after booting into Mac OS, you'll have to re-import your Mail.app emails, and then the next Time Machine run took ages, at least for me. My machine crashed twice while importing the mails before I increased fan speed - apparently there is still a heat problem. I also get some weird graphics errors (violet areas in windows, horizontal lines). I think I will get to know some more people in the Apple hotline shortly ...

Trying to control the heat issue, I switched from smcFanControl to Fan Control. smcFanControl only allows a user to set specific fan speeds (via presets), where Fan Control dynamically adjusts fan speed depending on current temperature (which in turn depends on work load). So if you have a large job running, it will dynamically increase fan speed a bit more than Mac OS would, to keep your computer a bit cooler. Nice.

I just installed all the new Windows XP patches in my virtual machine running parallels, including a bunch of hotfixes and service packs for Microsoft .NET.

As the download took quite a while, curious, I checked the size of the various .NET frameworks after installation. According to the Add/Remove programs dialog (no idea where these files really end up, so I can't check directly), .NET 2.0 and 3.0 consume ~280 MB and ~335 MB respectively, including the German language packs, each at ~100 MB. For the also installed .NET 1.1 there is no size given, but I'd guess it's not that much smaller then .NET 2.0, so about 150 MB should be a conservative guess.

So in summary, to run Microsoft .NET programs, I spend probable well above 750 MB of hard drive space. Compare to Java, where one version suffices due to backwards compatibility, where the JRE is 114 MB (and the JDK is larger, but not by much if I remember correctly). This is actually a pretty good argument for at least some investment in backwards compatibility.

I wonder what they include in those .NET packs? And what if they continue "innovation" (or whatever) at this pace? In 6 years since 2002 there are 3 incompatible version, does that mean we'll have to install 2 GB in 6 frameworks in another 6 years?

Online-Dictionary.nl is a nice service that translates, amongst others, from Dutch to German. I created a small bookmarklet that allows to quickly translate text on web pages. It's a direct rip-off from the LEO Dict bookmarklet.

First, drag this link: NL->DE to your address bar. Then select a word on a web page, and klick the link - this should open a window with the translation search. In case you didn't select anything, you will be prompted for a word to look up.

I'm trying to brush up my Dutch a bit, so apart from the great Woord van de dag service by the Niederlandistik (is that Dutchery in English? ;-)) at FU Berlin, I've started reading a bit in the Dutch Wikipedia, and this tool really helps.

GNOME Online Desktop, via Silvan, with screenshots and a tour available at RedHat Magazine.

This basically looks like a nice idea, going forward to really integrate desktop functionality with web based apps. However it feels somewhat backwards, with the desktop developers implementing lots of connectors to various web applications. Shouldn't it be the other way around?

What I'd envision is a desktop that tightly integrates with the web browser and provides a set of hooks for web applications to integrate with, something like a one-click installation of a small plugin that augments the desktop with functionality related to the web app.

This could be small JavaScript pieces or maybe even only XML configuration that tell the desktop where to search for documents/calendar events/IM conversations/..., how to integrate with IM, pull notifications and so on. That would make the system much more open - any website developer could nicely integrate his application, without relying on the GNOME developers to add his webapp to the desktop. Of course there are security issues with that, but they should be fixable.

I generally think there is much value in extended JavaScript access to the desktop. It is certainly dangerous and needs to be done right ™, but the possibilities are really cool - like access to calendars and address books. Mac OS nicely shows how this can work - they provide basic, central services like the address book and the calendar, and allow other applications to re-use the functionality, which is of huge value to users. This would also make it viable to write real applications for mobile devices (iPhone, Android) just through web pages and JavaScript. Users would need to be asked for permissions, just like they do it with HTML5's openDatabase offline storage, and a good user interface for that is crucial.

My guess is that even if the desktop fails to deliver such integration, the web applications will, sooner or later. Google already has all the right APIs in place, it's just lacking a proper model to share some information with some applications without exposing your whole online life to some foreign app (giving away your GMail username/password is not an option). So the question isn't whether this tight integration over services is happening, but rather whether the desktop will be part of it.

Some time ago I applied at a company that uses the "Taleo E-Recruitement" software. Taleo provides an ASP solution where - basically - HR people can post jobs and applicants can submit resumes. Gartner puts Taleo "in the leaders quadrant", as their website boasts. Once again, no idea on what Gartner judges (but some hints via Lars), but it's probably not related to the quality of the product.

I've rarely seen such a sucky web app, I thought they had died out somewhen around 2001. Search is pretty much broken, back button is broken, can't open pages in tabs, can't post links to jobs, it's integrated into the company's site in an iframe so that it takes a maximum of 1/3 of your screen, you need to create an account and will then be spammed with irrelevant job postings (need to login to turn them off, and the password recovery appears to be broken, too), etc. After you made it through the broken search and registration, you can submit your resume. First you can upload the CV, then you have to fix the automatically extracted name and address (why not just type it?!), then the system expects you to manually type your CV again in plain text, but please with formatting fixed. Then you'll have to re-enter all the information (work experience, education) from the CV in awkward HTML forms.

What could be simply writing a proper cover letter and sending it in an email including your CV is magically turned into a 1hr+ task, full of broken, annoying software and the constant fear that all your work will be eaten by another browser incompatibility on the last "wizard" page.

Taleo's website states that there is "Heightened competition for skilled workers." Yes, very much indeed. But why are you writing software that actively tries to keep people from applying to jobs then? It's like a specially designed filter that will drive off all the good people that don't need to put up with this stuff.

I'm currently finishing my Master's thesis (finally!), so I'm looking for a job. It's a bit weird though: there are apparently a gazillion of books on the market, telling you how to apply properly for job offerings, but from the job offerings I see, we desperately need literature on how to properly write a job offering.

You'll find hundreds of boring "J2EE/Hibernate/Spring/JUnit/$DB" ads, all listing a long number of technologies you ought to be familiar with (hint: good people will have no problems learning any technology!), and telling you effectively nothing about the company, the domain, or anything. Sucks.

And then you have all the ads that don't even meet the minimum standards - typos, duplicate copy/paste content, bad formatting, completely broken HTML. What are they thinking? Is that the impression you want to give, "we can't even produce proper job ads, come work for us"?

Once again random processes on my virtual server running this blog were killed, due to out of memory errors. This time, it actually led to some outage, because Apache was killed.

Normally, Apache servers the whole blog from a static cache, and only comments and new posts are handled by Rails, which makes the whole thing pretty fast - otherwise it would be unusable. Rendering single requests takes something close to seconds on my MacBook Pro, and thus much longer on the vserver. I have no idea why, and it's probably some bug in my own implementation, but introducing efficient caching was both easier and more interesting than performance tracing the rails code.

This is getting quite annoying. I have 128 MB guaranteed memory on the vserver, and this is simply not enough for Apache, SVN, Rails, and Tomcat. Interestingly, Tomcat is much less of a culprit than Rails, which consumes a lot more memory through the several process instances, as I found out earlier.

The bad thing is that there really is currently no way under Linux to find out which of your applications is actually using your memory, and how much of it. RSS, VSZ, "size" and others simply don't give any relevant insight into memory usage - the only thing you can probably do is look at memory consumption, start the app, and compare. Which is a pretty bad state, IMHO.

Lately I found that the /proc filesystem provides a 'smaps' file for each process, which contains its memory mappings, each explained with segment, rss, private and shared pages. This might actually lead to a useful memory analysis. One could probably write a simple tool that reports the private memory usage of each process, and accounts the memory usage through shared libraries to the apps using them. This should probably separate fixed cost (i.e., the shared memory consumed by running anything Ruby based) and dynamic costs (i.e., the added private memory for each Ruby/Rails instance).

I've started playing around a bit with a small Ruby script, and maybe if I have some spare time, I can turn it into something useful, though this will probably take a lot of learning about virtual memory in Linux.

Maybe I'm just using the wrong tools or operating systems, but I find it somehow depressing that we don't have a proper way of accounting memory to applications. It's really annoying that even today a badly written application that does something like while (true) malloc(...); can effectively bring down your whole system...

The requirements for XQuery 1.1 contain a MAY item called "higher order functions". I'm really fond of the idea of higher order functions in XQuery, and as there are currently no use cases for that, I'll contribute some in here:

Simple predicates

A simple use case is to pass a predicate to another function, as shown below:

(:: Selectively only copy those elements for which $pred returns true 
    TODO works recursively, but copys all attributes regardless of $pred
  :)
declare function local:selective-copy($nodes as element()*, $pred as function($node as element()) -> xs:boolean)
{
	for $n in $nodes
	return 
	  (: call $pred :)
	  if ($pred($n)) then element { node-name($n) } { $n/@*, local:selective-copy($n/*, $pred) }
	  else ()
};

declare function local:mypred($node as element()) as xs:boolean
{
  let $name = node-name($node)
  return namespace-uri-from-QName($name) = ('http://www.example.com/example', 'http://www.foo.com/bar')
};

let $xml :=
  <user xmlns="http://www.example.com/example">
    <name><first>Fritz</first> <last>Müller</last></name>
    <password xmlns="http://www.example.com/secure">vj/b5ZaUYQ6kU</password>
    <preferences> ... </preferences>
  </user>
(: user the curry operator & to get a function "handle" :)
let $predicate := &local:mypred
return local:selective-copy($xml, $predicate)

Currying

The next use case would be the natural extension - real currying of functions. local:selective-copy is as above, but we'll generalize our predicate a bit:

(: this function compares the namespace of a given node against a set of legal namespaces :)
declare function local:namespace-matches($namespaces as xs:anyURI*, $node as element()) as xs:boolean
{
  let $name = node-name($node)
  return namespace-uri-from-QName($name) = $namespaces
};

let $xml :=
  <user xmlns="http://www.example.com/example">
    <name><first>Fritz</first> <last>Müller</last></name>
    <password xmlns="http://www.example.com/secure">vj/b5ZaUYQ6kU</password>
    <preferences> ... </preferences>
  </user>
(: user the curry operator & to get a function "handle"
   we pass in a namespace to check against, and get an unary function in return :)
let $predicate := &local:mypred("http://www.example.com/example")
return local:selective-copy($xml, $predicate)

The & operator

The & operator ("curry operator") returns a handle to the function given by name, possibly setting values for parameters by specifying them in parentheses. This provides the reification for functions, i.e., the way from a function to a value.

Currying is only allowed from left to right, i.e., one cannot curry the second argument, but leave the first argument to a function unbound. I don't think this is a large restriction, and it makes the syntax much easier.

Calling function handles

The $someval(...) syntax allows straight forward calling of a function handle, so it is somewhat of the inverse of the & operator.

Types for higher order functions

I think the static typing feature of XQuery never really gained much traction. However it would of course be possible to introduce a new type, called "function()", and specify parameters and return values as shown above. I think it should be possible to type-check that, but I'm not an expert.

Implementation

I didn't implement this yet (I currently don't have access to an XQuery implementation), but it should not clash with any existing syntax, so from a grammar point of view it should be ok. It does require some changes to the runtime system, but that shouldn't be too difficult, IMHO.

The nice thing about higher order functions is that they can allow some method of dynamic dispatch. That is, they allow to write programs that decide at runtime which code is going to be executed, in an elegant way.

This is of course not complete. It doesn't support a terse syntax for lambdas, which would also be nice (not having to declare all those pesky one-line functions). XQuery should also have something like a "fn:resolve-function($name, $argc)" that provides dynamic access to the curry operator by specifying the QName and argument count of the function.

I think the example shows that this little extension can get you a lot of nice functionality and a lot less typing. Please leave a comment with your opinion!