Articles in the Nepomuk category

Finishing Touches and GMail

Published on jeu 29 août 2013 in Nepomuk, (Comments)

It has been a long time since my last blog post, but that doesn't mean nothing new has happened. In addition to fine-tuning the new Nepomuk Query Parser, I also try to write a Konqueror plugin that detects emails on common webmails and that index them in Nepomuk.

Fine-Tuning of the Query Parser

At the beginning of August, the query parser was already in quite a good shape. It was able to parse many queries in English, and I translated it into French the check that everything goes well. The first thing I discovered is that even if the translation operation was easy, it could be even better.

The problem was that the pattern matcher syntax was not adapted to languages with rich "declination". In English, adjectives are always spelled the same, independently of the words around them. In French, adjectives can have four forms, as they have a genre (English people always find funny that "A car is a lady in French"), and a number (the singular form is different from the plural one).

An example shows the problem better: consider the "modified|changed at %1" pattern, that matches "modified at 3.20pm" for instance. In English, two different adjectives are given, and the pattern works well for "files modified at ..." and "file modified at ...". In French, the pattern needs to accommodate the four forms that each adjective can take. It becomes "modifié|modifiée|modifiés|modifiées|changé|changée|changés|changées à %1". It's quite long, isn't it ?

The first step towards a solution is to allow regular expressions in patterns. The pattern in French now becomes "modifiée?s?|changée?s? à %1", its length is again under control. Regular expressions is the best I can give to translators, even if I think that many languages are still too complex to have each word expressed using a single expression. That is not a problem as it is still possible to use several expressions in a single pattern, so at worst the translators just need to write the 60 forms of their words, separated by pipes.

The problem of this solution is that the pattern is shown to the user, as you can see on the above image. Patterns that don't use regular expressions are nice to display (they are words separated by pipes), but regular expressions are way more difficult to understand. Now that patterns are shorter but not user-friendly anymore, I needed to find a way not to have them displayed.

This is done in two steps. The first one is that the user input is displayed when possible. If the user has typed "Modified at", the auto-completion box will show "Modified at", not "modified|created at|on". This already simplifies the auto-completion box, and has the nice property of respecting the case and the exact spelling the user used.

The second step is to display only one possible literal per term after the user query. If the pattern is "not touched|modified|changed since|after %1" and the user entered "not modified", the completion proposal will be "not modified since [something]". Translators can now hide the regular expressions: they just have to start their list of regular expressions with a simple nice word, that serves as a "title": "modifiée?s?|changée?s? à %1" becomes "modifié|modifiée?s?|changée?s? à %1", and the auto-completion popup will only display "modifié", never the following regular expressions.

Extracting Information from Web Pages using Javascript

By "parsing", I mean "recognizing the content of specific elements of a page". At the beginning of the month, I started to experiment with extracting emails from webmails and indexing them in Nepomuk. The first extractor I made is the one for Yahoo! Mail Neo (the French version). This extractor works very well.

After that, I wanted to quickly implement a parser for GMail and Outlook, two of the most used webmails. I started with GMail, and discovered that the current infrastructure did not allow to parse it. The problem is that this webmail is very complex. For instance, an email can be displayed on its own page (new messages from unknown contacts) or in a discussion (when you reply to someone). The discussion is especially difficult to parse as information about the emails is scattered on the whole page: the content of the messages is in the discussion itself, but the meta-data (date, sender, title) is contained in elements very far from the messages in the DOM tree.

I therefore decided to replace my own JSON-based syntax with full-fledged Javascript. QtScript made that very easy, and in one day, I had the Yahoo! Mail parser rewritten in Javascript:

({
    urls: ["*.mail.yahoo.com/neo/*"],

    emails: function(document) {
        var content_nodes = document.findAll("div.message.content");
        var mails = new Array();

        var field = function(parent, name, attribute) {
            var nodes = parent.findAll("div.info.expanded ul[aria-labelledby=\"hdr-" + name + "\"] a");

            if (attribute == "") {
                return contentOf(nodes);
            } else {
                return attributeOf(nodes, attribute);
            }
        }

        for (var i=0; i<content_nodes.length; ++i) {
            var content_node = content_nodes[i];

            var date = content_node.findFirst("div.info div.date");
            var title = content_node.findFirst("div.info h3 span.txt");
            var content = content_node.findFirst("div.msg-body");

            if (date == null || title == null || content == null) {
                continue;
            }

            mails[i] = {
                title: title.content(),
                content: content.content(),
                date: stringToDate(date.content(), "ddd, d MMM. yyyy à h:mm", "fr"),
                from_name: field(content_node, "from", ""),
                from_addr: field(content_node, "from", "title"),
                to_name: field(content_node, "to", ""),
                to_addr: field(content_node, "to", "title"),
                cc_name: field(content_node, "cc", ""),
                cc_addr: field(content_node, "cc", "title")
            }
        }

        return mails;
    }
})

The JS rules can use everything exposed by QtScript (I think that it is standard core EcmaScript), plus some helper functions implemented in C++. The DOM Elements are represented by JSElement objects (each JSElement being either a JSElementKHTML or a JSElementWebkit), and they have a small set of methods:

tagName() that returns the tag name of the element
content() that returns the text content of the element
attribute(name) to get the value of an attribute
findAll(selector) that returns the list of all child elements matching a CSS rule
findFirst(selector) that returns the first element returned by findAll, or null if the list of matched elements is empty

contentOf takes a list of elements and returns a list containing their content, and attributeOf does the same for an attribute. Finally, stringToDate(string, format, lang) converts a string to a date using a specified locale. Internally, QLocale::toDateTime is used.

Detecting Changes in the DOM Tree of a Document

After having implemented rules written in Javascript, I decided to try again to parse GMail. Its rule is a bit more complex than the one of Yahoo, but is was possible to implement.

The problem is that when I tried to test my rule, nothing happened. Why ? Because the Konqueror plugin needs to detect when the DOM tree of a page changes. This notification triggers a re-parse of the page, and the parsing rules can detect potential new emails. In Yahoo! Mail, the DOM changes are correctly detected: they are preceded by a network request that can be detected by using the KParts::ReadOnlyPart::started(KJob *) signal. GMail does not trigger this signal, though. By sniffing my network, I discovered that GMail also performs network queries just before updating its DOM tree, but these queries are not detected. I don't know why and I have asked on the kde-devel mailing-list. If you have a solution for reliably detecting DOM tree changes, you will have my eternal gratitude.

I will try to solve this "GMail problem" in the coming days. After that, I will implement rules for Outlook and Roundcube. This will allow me to check that the current Javascript-based infrastructure is right, and that I can proceed to the development of a Firefox plugin. If you have ideas regarding the Nepomuk Query Parser or find bugs in it, don't hesitate to tell me as I very muck like to work on it, and I want to make it the best possible.

A Nepomuk Integration Plugin for Konqueror

Published on mer 07 août 2013 in Nepomuk, (Comments)

Last week, I presented an idea about indexing messages from webmails in Nepomuk. The summary of this idea is to implement a browser extension for Firefox, Chrome and Konqueror. This extension parses the DOM tree of every page visited by the user that belongs to a webmail. When e-mails are found, they are extracted and stored in a temporary file. Nepomukfileindexer then quicks in and indexes these e-mails in the Nepomuk database.

The reason why I started this experiment is because many users use webmails instead of mail client programs like KMail or Thunderbird. The information about these emails cannot therefore be indexed by Nepomuk. As there are many different webmails, the biggest part of the project is to implement parsing rules for each of them. The browser plugin itself is quite small.

Extracting Information From the DOM Tree

Last week, I implemented the file indexers that are responsible of indexing mbox files and MIME files into the Nepomuk database. These indexers are now in a fairly good shape. This week, I started to implement the browser plugins. I wanted to start with Firefox, as it is the most used web browser, but my experience with Javascript is quite limited. I therefore started with Konqueror, more on that later.

The browser plugin reacts on DOM tree changes. When the DOM tree of a page has changed, the URL of the page is used to determine on what webmail the user is. If rules exist for this webmail, they are applied.

A rule is a simple JSON file that describes how to extract information from the DOM tree. I use JSON because parsing it in C++ is easily done using QJSON, and parsing JSON in Javascript is as easy as an eval call. The DOM is read using CSS selectors, that is to say selectors as used in CSS style sheets (div.test matches every <div> tag that has the class "test"). The first rule I developed is the one for Yahoo! Mail in French (Yahoo! Mail has the bad idea of displaying a strongly localized date-time, and the ISO-formatted date-time is found nowhere in the document):

{
    "urls": ["*fr*.mail.yahoo.com/neo/*"],
    "date_format": "ddd, d MMM. yyyy à h:mm",
    "emails": {
        "node": "div.message.content",
        "date": {
            "node": "div.info div.date"
        },
        "title": {
            "node": "div.info h3 span.txt"
        },
        ...
        "content": {
            "node": "div.msg-body"
        }
    }
}

Applying this JSON pattern against the DOM tree yields a tree of QVariant values. Each QVariant is either a list, a map or a string. The tree looks like that:

emails = [
    {"date": "Mar, 7 aoû. 2013 à 13:12,
     "title": "This is a mail",
     "content": "Hi\n\nThis is an email!\n\nCheers"
    }
]

This data is stored into a MIME file using the wonderful KMIME::Message class. The MIME file is stored in /tmp/kde-username/.

The Konqueror plugin

The source code is available my nepomukintegration scratch repository.

The Konqueror plugin is currently only 300-lines long. QJSON does all the JSON parsing, KMIME::Message writes the MIME files. What is left to do in C++ is matching the patterns against the DOM tree. Here, Konqueror acts as two web browsers: you can choose to display web pages using either KHTML or kdewebkit. The two engines do not share the same structure. KHTML is based on KDE specific DOM classes, and kdewebkit uses a QWebView.

Matching a pattern therefore needs to be done for the two engines. Instead of duplicating all the code, I used a big template function. When the APIs of the two engines are different, I implement utility functions with an overload for each engine:

static QString elementContent(const DOM::Element &element)
{
    return element.textContent().string();
}

static QString elementContent(const QWebElement &element)
{
    return element.toPlainText();
}

The template functions only uses the utility functions, and the compiler chooses the right ones.

Status

The Konqueror plugin is now quite usable. It works, it does not crash, and my emails are found. In fact, my emails get properly extracted to temporary files, and running nepomukindexer correctly indexes the data in my Nepomuk database. The plugin can therefore be considered as starting to be functional.

Indexing mails is one thing, contacts is another one. "Simple" contacts are already indexed: if the sender of an e-mail (identified by its name and its e-mail address) is not already present in the Nepomuk database, a new contact is added. By parsing contact-specific pages of webmails, it should be possible to extract more information about the contacts, like their birthday, gender, etc. This information will be stored in vCard files, indexed by the vCard file indexer I developed last Sunday. Another data I would like to index is history.

You may be wondering why I'm storing the extracted information in files instead of storing it directly in Nepomuk. In fact, Konqueror is a KDE application and has access to Nepomuk. The reason is that I don't want to duplicate work. Nepomukindexer already handles the cleanup of existing meta-data, and my email indexer ensures that an email never gets indexed more than one time. Moreover, the extension needs to be quite "cross-browser compatible", and Firefox and Chrome have no access to Nepomuk (Nepomuk requires a Qt event loop, and integrating the Qt event loop in GTK-based applications is tricky, see what the GTK Qt style has to do).

Extracting Information from the Web

Published on ven 02 août 2013 in Nepomuk, (Comments)
Google has just sent me a mail saying that I passed the midterm evaluations of the Google Summer of Code! Yay! I want to thank everybody that helps me in my GSoC project, everyone on the Nepomuk mailing list, and more generally everybody part of the KDE community. Everyone is very friendly, and working on KDE is a very enjoying experience!

As the query parser and the query builder widget come close to being usable (I have still bugs to fix, and things to change in order to make the query parser more localization-friendly), I'm beginning to think about other things on which I can work. One of these thinks is storing in Nepomuk information that comes from the user's web browsing.

The Problem

Nepomuk works really well, and now has a good query parser and a fairly nice query builder widget (I think). You can index your files, meta-data about them is extracted and stored in the database, and you can query this database using Dolphin or KRunner. The problem is that this indexing is currently limited to files.

If you use KMail or any Akonadi-based application, your e-mails and contacts are also stored in Nepomuk, ready to be queried using queries like "mails from John". The problem is that KMail is not used by every KDE user, as some of them prefer other mail clients (that don't use Akonadi). In fact, the majority of the KDE users use webmails, and their instant messaging applications are not KDE ones. Furthermore, many Linux distributions install Firefox as the default web browser. Most of our users therefore don't enjoy the Nepomuk integration that KMail, Kopete and other applications like that provide.

Indexing Information from the Web

My idea is to index in Nepomuk information found on the web, but not automatically, not by using a web crawler. The user itself will provide this information by browsing the web. The idea is to implement a Firefox add-on (and after that a Chrome add-on, as Chrome is also widely used and it seems that Rekonq supports or will support Chrome add-ons) that extracts information like that (if the user allows it to do so) :
- The fact that a page has been visited (I just have to check that Nepomuk has an ontology for storing web history)
- If the page is recognized as being one of a known web-mail, its DOM structure is explored to find information about a displayed mail or contact
- When downloading files, the URL from which it is downloaded can be stored in Nepomuk (there is already a Firefox add-on that does that)
More File Indexers

As Vishesh Handa once explained in a blog post, Nepomuk uses file indexers to extract information about files (line count, video format, information about photographs, etc). Adding a file indexer is very easy, even complex ones are only a couple hundred lines long. The file indexer gets a file URL, passes it to a library (ffmpeg, taglib, etc), gets information and stores it into Nepomuk.

For my new experiment of storing web information in Nepomuk, I have to develop three new file indexers:
- An e-mail indexer. It uses the wonderful KMIME and KMBOX libraries (part of kdepimlibs) to open mbox files and to index the mails it contains. For maildir directories, the e-mails are directly indexed using only KMIME. The indexing consists of storing in Nepomuk the plain text content of the mails and their headers (sender, receiver, etc). Senders and receivers are also used to set the e-mail address of contacts. This is interesting because the office extractors (MS Office and ODF) already create contacts for the authors of your documents. This indexer can therefore add an e-mail address information to these contacts.
- A vCard indexer, that uses the KABC library of kdepimlibs to extract detailed contact information from vCard files (name, street address, phone number, etc).
- An indexer for nepomuk-specific file formats. For instance, a file format can represent associations between downloaded files and their original URL. Another one can represent a piece of navigation history.
Two of these indexers use the wonderful libraries developed by the kdepimlibs developers. They are rock-solid and implement very complex standards (MIME files are hairy to handle). I want to thank them very much for these libraries that are a pleasure to use:
1 2 3 4 5 6 7 8 9
KMime::Message message; message.setContent(raw_data_from_file); message.parse(); resource.addProperty( NMO::plainTextMessageContent(), message.decodedText(true, true) );
Bridging the Web and Nepomuk

Why do I use file indexers to index information coming from the web ? Because browser extensions are limited. Firefox and Chrome are both based on GTK+, and integrating Nepomuk and its Qt event loop would be terribly difficult. Moreover, Chrome does not allow native extensions at all (I'm not sure about that, my information comes from very early Chrome versions).

Browser extensions can perform HTTP queries, though, but I do not want to develop an HTTP server that would listen to POST request to create Nepomuk resources. I have already developed an HTTP/1.1 parser (with support of chunked encoding etc), but it is buggy and fairly complex.

The solution lives in files: Firefox and Chrome extensions both are allowed to create local files (it's easy to do in Firefox, less so in Chrome, but possible). So, when a piece of information is extracted from a web page, the extension only needs to format it according to a web standard (MIME, mbox, vCard, etc) or a Nepomuk-specific one (for history and downloaded files), and then to store it in a temporary directory monitored by nepomukfilewatch. nepomukfilewatch is a service that indexes files when they are created or modified, so it is possible to detect these new files and to index them. The bridge is in place.

Using file indexers also has a nice side effect: users using Thunderbird/Mutt/etc can point nepomukfilewatch to their e-mail directories (it is a simple as checking them in the KCM module of Nepomuk), and have their e-mails and contacts indexed by the e-mail indexer. Akonadi users don't have to do anything as Akonadi already stores information in Nepomuk. And as the e-mail indexer skips e-mails already stored in Nepomuk (a mbox file only grows and can contain thousands of mails, it would be way too slow to re-index them at every run), if a mail folder is indexed by both the e-mail indexer and Akonadi, there will be no conflict.

Current status

As this projects has not (yet) been approved by the Nepomuk developers, all the code still lives in the steckdenis-webindexers branch of my nepomuk-core git repository. The code is still minimal, but the e-mail indexer is already there. It can index mbox files and plain e-mails stored in maildir directories. I plan to implement a vCard indexer shortly.

As this project is in a very early stage, I encourage you to comment on it and to give me your impressions. Do you think that this architecture is right ? Is my use-case a good one (the fact that most users nowadays use webmails) ?
Translations and Better Auto-Completion

Published on sam 27 juillet 2013 in Nepomuk, (Comments)

This blog post presents several small steps that made the new Nepomuk query parser closer to be useful for every user. The most important one is that its localization features work, the other is that the auto-completion now is more clean and elegant.

Before speaking of new features, a good news: my work now lives in the official Nepomuk repositories! I got a KDE Developer Account, and you can find everything in the steckdenis-gsoc2013 branches of nepomuk-core and nepomuk-widgets.

The Query Parser in French

Even if I don't live in France (I live in Brussels, Belgium), my native language is French. In fact, Belgium is a country where half the population speaks Dutch and the other half French. There is also a group of people speaking German. In Brussels, the capital, about 80% of the people speak French.

That's for the geolinguistics. What interests most readers is that I translated my Nepomuk query parser in French. This was quite easy: every translatable string is wrapped into a i18nc() call, and adding a small Messages.sh file in the Git repository enabled me to extract these messages using a script of the KDE Localization project. The translations were extracted into a .pot file, that I just had to edit using Lokalize. When every string was translated, I saved the file and installed it on the right place on my system.

The result is shown in the following screenshot, that will surely look familiar to French-speaking people:

The .pot are not in the Git repository because it does not belong there. When my work will be merged into the master branch of nepomuk-core, scripty (the script that updates translations on the KDE Git and SVN repositories) will extract the translatable strings and will put them on the SVN repository used by the KDE translators. Each language will then have its own translation of the parser.

Simplifying the Auto-Completion Box

The auto-completion box as shown in this blog post was made of pages. The first one displayed patterns (what the user can enter and its meaning), the second one was used for strings (tag names or contacts), and the third one was a QCalendarWidget, that allowed the user to select a day.

Even if this worked, there were several problems in this approach. The first one is that the code was not very nice, there was duplication between the pages. Another problem was that the QCalendarWidget may not be compatible with other calendar systems. Finally, as you can see at the beginning of the blog post, the widget was not very nice. Sure, the page displaying patterns had charm, but the list of strings looked too simple.

Another problem, that cannot be shown using screenshots, is that getting a list of strings is very difficult. As this list of strings replaces the list of patterns, there must be only one pattern in the list. This means that you won't get the list of your tags if your query is even slightly ambiguous, and it is easy to be ambiguous. For instance, "sent to" shows three patterns: "sent to [contact]", "to [contact]" (ignoring the "sent") and "sent [date-time]", as the pattern matched does not know that "to" will never be a valid date-time. In this case, even if you know what you want, the query builder with not show you anything.

The solution I adopted is to get rid of the pages. Now, the completion box only shows patterns (this was the nicest page). When values can be auto-completed, they are so directly in the pattern list. For instance, if you type "sent to Nep", the auto-completion box will show "sent to 'Nepomuk Mailing List'" and "to 'Nepomuk Mailing List'". This way, the user gets value proposals even if its query is slightly ambiguous. And this is nicer than a big list of strings without any formatting:

This change also removed hundreds of source lines of code. I'm always happy to improve things while removing lines. The only "regression" is that the calendar widget is gone, and I still need to find a way to tell the user that he or she can enter human date-times ("if you want to open a calendar to find what is the first Monday of next month, just type 'first Monday of next month'").

When KRunner Works out of the Box

After having integrated the query builder widget into Dolphin, I wanted to see if there was something needed to parse Nepomuk queries in KRunner. Before cloning any git repository, I tested my distribution KRunner. As I install my development libnepomukcore system-wide (this allows me to check that everything is binary-compatible), entering a query in KRunner yielded:

What a nice surprise! Nothing to do, KRunner is able to use my new parser without any modification. I thought that working on an already well-known library is a very pleasant experience, because you don't improve one application, but every application that uses the library. Furthermore, the fact that KRunner did not crash means that my work is fully binary compatible with the old query parser (as I reused its .h header, I was quite confident in that).

Other bugs fixed

The other changes don't deserve a screenshot. For instance, I fixed how date-time intervals are parsed (there was a typo that made weeks one-day long, and another one that inverted December and September) and I improved how relative weeks are handled. For instance, "second week of next month" now works as expected. This fix also simplified the code.

The query builder widget was also slightly improved. Its sizing is now more consistent, and I removed the sizeHint() method (that I simply ignore now). All in all, the parser and the query builder now work way better. I also added Doxygen comments to CompletionProposal. I still have to do that for the query builder widget itself.

Some days ago, I wondered if implementing a Firefox extension that recognized e-mails and contacts in webmails was a good idea. I read documentation about that and it seems possible, users using webmails will be able to enjoy the Nepomuk features. Before doing that, I will first check that nobody has done something like that. The KDE QuickGit for instance lists interesting repositories, like nepomuk-web-extractor.

I also add that if you have queries that you think the parser should be able to parse, don't hesitate to tell me. There may be whole classes of queries that I haven't thought of.
Integrating the Query Builder into Dolphin

Published on mer 24 juillet 2013 in Nepomuk, (Comments)
Something that works is good, but something that works great is better. These past days, I fixed many bugs and imperfections of the new Nepomuk Query Parser and the Query Builder widget. I will summarize theses fixes shortly, then will come the screenshots.

Syntax-highlighting, one more time

Before my holidays, I implemented the Query Builder widget as a syntax-highlighted QPlainTextEdit. This was easy and fun, and the result looked quite nice. You can find the whole story here.

After that, I decided to rewrite the implementation from scratch to give it a different look. I wanted to have boxes around parsed terms, with a small cross to remove them. After some days, the widget looked like that:

This widget was really nice, but its implementation did not seem very clean. Furthermore, the widget was made of a complex tree of line edits, buttons and widgets responsible for drawing the boxes. Handling the cursor position (and the text itself) was a bit hackish, and some things did not work well (you were unable to select all the text of the widget, for instance, and focus handling was a bit special).

On Sunday, I thought of a new implementation that has the simplicity of the syntax-highlighted QPlainTextEdit, with the look of the complex widget tree.

At the start of the day, I wanted to use a QTextEdit with a custom style sheet and custom HTML. My plan was to use the HTML renderer of Qt to draw the boxes around the parsed terms, and to use the Unicode "×" character to provide the cross. I wanted to implement the closing cross using a custom mousePressEvent method and something like characterAtMousePosition() == '×'.

Finally, I discovered that the HTML engine of Qt does not support rounded corners (and does not seem to support border colors at all). So, I decided to do all the drawing myself, in a paintEvent method. I tried to subclass QLineEdit, but this class does not allow me to map a character position to a x coordinate. Finally, I used QPlainTextEdit, because QTextDocument allows me to map a character position to a x coordinate:
1 2
QTextLine line = document()->findBlock(0).layout()->lineAt(0); int x_pos = line.cursorToX(char_index);
Using this, I'm able to easily draw boxes around terms, and this is fast. The only problem is that there is no crosses. I first wanted to add " ×" after each box, but this would have required to change the positions of the boxes. If a box spans characters 2 to 6 without a cross, it needs to be extended to character 8 with a space and a cross at its end. Furthermore, the following boxes need to be adjusted 2 characters to the right. This would have been too complicated for a feature not really needed.

With the third version of the query builder in place, I was able to fix many bugs. Text selection now works correctly and as expected, and the auto-completion widget was enhanced. It preselects completions based on what the user types (if I type "tagged as Im" and I have a tag named "Important", it will be pre-selected), and many little details like size hints are now saner.

Playing with Dolphin

Yesterday, I wrote a small patch for Dolphin to integrate the query builder in it. When Dolphin is built with Nepomuk enabled, you can enter user queries with the help of the query builder widget. The result is quite nice and I hope that this patch will be merged in the next KDE release. I don't post it yet on Reviewboard because the API of the query builder is not yet fixed.

The next steps are now to integrate the query builder in more applications (maybe Plasma Active), and to fix even more bugs. I also have to document the public API of my code.
The Journey to Auto-Completion

Published on ven 19 juillet 2013 in Nepomuk, (Comments)

Sometimes, features are simple to describe, and their implementation also ends up being simple and small. However, the reflection needed to come with the implementation can be quite substantial. This blog post relates my journey to the auto-completion feature of the Nepomuk query parser, a feature described as "show to the user want he/she wants" and implemented in less than 1000 lines (empty lines and license headers included), but that took a whole week to implement.

Features

Before explaining how the auto-completion works, let's start with the parsing itself. As described in previous blog posts, the Nepomuk query parser tries to find in the user query lists of contiguous terms that match a certain pattern. For instance, if the query is "mails sent to Jimmy", "sent to Jimmy" is matched against "sent to %1".

The idea of the auto-completer is to use these patterns. What the user wants is to be able to type the beginning of a pattern, and to have all of it auto-completed. When what the user has entered corresponds to the beginning of a pattern, this pattern is shown as an auto-completion proposal. When the user selects a proposal, the pattern is put into the query builder and the user can continue its input.

When there is only one possible pattern, that is to say when the user is unambiguous about what he or she wants, then a more intelligent auto-completion can be used. As you can see, the auto-completer knows the type of the value associated with a pattern. "tagged as %1" for instance expects that the user will enter a valid tag name. So, when there is only one possible pattern, a one-item list is not shown (it would be useless), but instead an auto-completion box adapted to the type is displayed.

You see that tag names and date-times use a different auto-completion widget. As this is still preliminary work, the list is not sorted, there is no search interface (it would be difficult to do in a popup, by the way), and contacts are not yet handled. This is not because it is technically difficult (contacts can be enumerated just like tags), but because I don't know how Akonadi stores contacts in Nepomuk, if it does, and if it is what most of our users want (the majority of users nowadays use web-based email clients and proprietary instant messaging services).

Auto-Completion in the Parser

My work on auto-completion has been split between the parser and the query builder widget. The parser is responsible for finding the auto-completion proposals. It knows the list of all the possible patterns, their types, etc.

The first step was to add a CompletionProposal class (in the Nepomuk2::Query namespace). This class simply stores a a pattern (a string list of terms that must be matched), a type (whether the user needs to enter a tag name, a date-time, or a simple string), a textual and localized description, and positional information. This positional information tells what part of the user query needs to be replaced with the auto-completion if the user selects it. For instance, an auto-completion for "sent to" in "mails sent" will preserve "mails", but will replace "sent" by "sent to |", with the bar representing the text cursor.

As auto-completion proposals are dependent of where the text cursor is, I've added a third overload of QueryParser::parse. This method takes a user query, flags, and the position of the cursor that will be used to find auto-completion proposals. It is a bit special that the parser has to know the position of the cursor, but it greatly simplifies things and avoids generating too many auto-completion proposals, that will later be rejected because they are too far from the cursor.

With all that done, the actual implementation is quite simple, less than 40 lines long. When the pattern matcher (the class responsible for matching "sent to Jimmy" against "sent to %1") is done with a pattern, it checks if the cursor is contained in the pattern, or very near. If it is the case, a new completion proposal is created.

Auto-Completion in the Query Builder

The implementation in the query builder widget is a bit bigger, because I had to add an auto-completion popup (the thing that displays the auto-completion proposals and looks like the drop-down menu of a combo box) and the widgets used to display the proposals, the tag list and the calendar (the calendar itself is a QCalendarWidget).

After the user query has been parsed (with the position of the text cursor passed to the query parser), the query builder widget enumerates the auto-completion proposals. If the list is empty, the auto-completer is hidden. If the list contains only one entry, then a datatype-specific completer is shown. In all the other cases, each completion proposal is added to the completer, and finally displayed in a QListWidget with custom item widgets (to have the description in bold and the pattern below).

The end result is not so bad but not yet perfect. For instance, there is no border around the drop-down completion box. I've looked at the QCompleter and QComboBox code but I have not found how they manage to have rounded styled corners and a small 3D effect.

When the user selects a value (a tag name, or a date-time), the value is put into the query builder widget:

Future plans

Having the auto-completion in place is a nice thing because my GSoC proposal is now "bullet-point complete". That means that I proposed a query parser able to parse queries in near-natural language, and a query builder widget with syntax-highlighting and auto-completion. These three features are now present.

But even if they are present, I no way near to the end of my GSoC. Many things need to be polished. For instance, I'm not quite satisfied of my "grouped line edit". Some things don't work exactly as I want, and the architecture is a bit special.

There are also small bugs in the parser itself that need to be resolved, and the auto-completer is not very nice. The mockups of Ivan also contain a nice feature I would never have thought of: the translation of the parsed query back to natural language. That means than in his mockup, when he enters "music +instrumental", the auto-completer displays "looking for music tagged as instrumental". I think it is a good idea to show to the user what the query really means.

All in all, there is still work to do and I will fix as many bugs as I can in the coming weeks. My goal is to have a very nice query builder widget, that works well and that has a code of great quality.