Latest posts

• GroupedLineEdit Reused in Subsurface

  Published on dim 16 février 2014 in Nepomuk, (Comments)

  Hi,

  It has been a long time since I last blogged about the Nepomuk user-query parser that I developed during the summer. The reason is that the university required all my time. Here is a quick blog post that summarizes what happened these last months.

  Merging the Query Parser

  For those who missed the news, Nepomuk has been "abandoned" and renamed to Baloo. The reason is that the Nepomuk developers thought that using RDF (a complete and complex system of ontologies, that require specialized databases like Virtuoso) wasn't the right thing to do in order to provide what the user want, that is to say a powerful yet lightweight search engine. RDF allows many things, but most of the users just want to look for documents matching a specific string.

  I cannot better explain the reasons, but you can read this mailing-list thread if you want more accurate details.

  What is important is that Nepomuk has been rewritten, and that breaks my user-query parser. Fortunately, the parser is pretty much self-contained, and the algorithms used don't need to change, but the way the parser provides its results to Nepomuk has changed. In Baloo, queries seem to be simpler (for what I've seen) and are not built out of trees anymore.

  In Nepomuk, a query was a big AND, containing ORs, other ANDs and actuel comparisons. Getting every document containing "test" and tagged as "important" was done using a query like AND(tagged_as(important), containing(test)) (this syntax is not used anywhere in Nepomuk but represents in-memory Nepomuk2::Term subclasses). In Baloo, for what I've seen in the source code, a query is more "fixed". Every query consists of search terms ("test" in my example) and filters (date-time filters, maybe also tags, etc). There does not seem to be any recursive data structure. Because of that, the user-query parser cannot simply convert its abstract syntax tree to a tree of terms (a very simple operation), but needs to carefully analyze the user-query in order to produce the Baloo query (or queries) that will provide the expected results.

  I'm still thinking about all that and I will contact the Nepomuk developers for advice, because porting the user-query parser to Baloo seems not to be that complex, and seems very interesting.

  GroupedLineEdit gets used by Subsurface

  Free Software and its communities are full of great surprises. One of them was when Aaron Seigo quoted me on Google+, or when the Nepomuk file indexer for MS Office 2003 files, that I developed in October, was mentioned in the KDE Commit Digest.

  Several weeks ago, just before the FOSDEM, the Subsurface team announced that the project was ported to Qt. Subsurface is a divelog program started by Linus Torvalds and now developed by a fair number of contributors, some of them being well-known kernel hackers. When Subsurface 4 was announced, I read the news (even though I don't dive, I wanted to see how a GTK+ program could be ported to Qt and what that brings as new/improved features), and I discovered a very intriguing screenshot in the user manual :

  

  Have you noticed that there is a "Tags" field? And that it contains tags highlighted in colors? When I noticed that, I thought that Linus (or someone else) had developed exactly the widget that I needed during the summer, the widget that is used to highlight query terms in user-queries, as shown in this image :

  

  I downloaded the source code of Subsurface and started to look after this widget that does exactly what I want. Finally, I found it, and I was very surprised to see that it was actually my GroupedLineEdit widget that Subsurface used! I'm very pleased to see that something that took me several weeks to develop (the widget is less than 200-lines long, but I needed a great deal of time to figure out what was the nicest way to highlight text, and how to implement it properly) is used, and used in a program started by nobody else than Linus Torvalds.

  So, if anyone wants to develop something : don't hesitate to do it! Even if it is a small widget or a toy application, what you develop may be of use to someone else, and it's always a great pleasure to see that code we have developed gets used by other developers.

• The End of the Google Summer of Code

  Published on mer 25 septembre 2013 in Nepomuk, (Comments)

  The GSoC coding period has ended last Monday. In this blog post, I will summarize what I have done during the summer, to give a general idea of what you can expect from a future version of Nepomuk, if my work is accepted by the Nepomuk maintainers.

  The source code of the first two projects is available in the steckdenis-gsoc2013 branches of the nepomuk-core and nepomuk-widgets Git repositories. The browser plugin can be found in the scratch/dsteckelmacher/nepomukintegration.git scratch repository.

  The Nepomuk Query Parser

  The first part of my GSoC project was to implement a new user query parser for Nepomuk. This parser is responsible for translating queries entered by the user into a formal representation suitable for querying the Nepomuk database.

  Project Goals

  The current version of the parser (that is used for years now) is based on regular expressions. It recognizes a language close to what you can use on Google ("foo -bar title:something"). At the beginning of the project, Vishesh Handa explained me that this parser has limitations (it is not able to parse file sizes, and its parsing of date-time is a bit special, the parser tries many formats in an arbitrary and locale-independent order), and that its syntax is not very user-friendly. We agreed on implementing a parser that has a syntax more close to natural language, a syntax simple to understand and that doesn't require the user to read a big book before being able to perform queries.

  The "new" query parser, as I name it, is based on pattern-matching. My first idea was to implement a real parser using state-of-the-art parsing techniques like Abstract Syntax Trees and a formal grammar, but these kinds of languages are not very user-friendly and are more suitable for programming languages, or anything that must have a precise sense of correctness and a defined meaning. The new parser aims for being user-friendly and accepting anything the user can enter, without the possibility of a parse error.

  Recognizing Human Writing

  During the spring, I developed a human date-time parser, that is to say a parser for date-times expressed like "first week of last month". Parsing such strings can nearly be considered as artificial intelligence, a field that greatly interests me. The way a human understands what is said or written always bothered me.

  The literature regarding this subject is very rich, and complex algorithms can be used. More simple ones are possible when less precision is needed, and parsing search queries doesn't need a huge amount of precision. This allowed me to simply consider that as humans learn a language by memorizing the signification of words, the structure of sentences, and how patterns map to a signification ("X's Y" = "the Y of X" = Y.owner: X, X.has: Y), a computer could do the same.

  I therefore implemented the date-time parser as a pattern-matcher. The parser considers the full string, and tries to find information in it. When words are matched against patterns, they are removed from the string. This way, the string contains less and less information, and the parsing stops when everything is removed. For instance, "first Monday of last month" contains two pieces of information : we talk about the first Monday of something, and this something likely is last month. The parser simply ignores everything it doesn't understand. A human confronted to a language he or she doesn't completely understand also works this way : a complete sentence is understood as "Warning, ... due to works ... metro stopped ... we apologize ...".

  Matching patterns

  As the date-time parser worked fairly well, I used the same algorithm for the Nepomuk query parser. The user query is split by words (if the user's locale doesn't separate words, the query is split by characters), that are used to build Nepomuk terms. At this step, the query looks for documents containing everything you typed. The query is already valid.

  Next, the parser tries to recognize patterns, that are fully translatable (the order of the words can be rearranged, synonyms are handled, and multiple patterns can be associated with the same meaning). When a pattern is recognized, placeholders are used and the pattern is replaced by its parsed meaning. For instance, "tagged as X" is recognized in "Nepomuk, tagged as Important", and is replaced by a property comparison.

  Because matched patterns are replaced with their parsed value (and not discarded as in the date-time parser), patterns can be nested. For instance, "size > 2M" contains three patterns: "2M" that is replaced by "filesize=2000000", "> comparison" that is replaced by "filesize>=2000000", and finally "size comparison" that is replaced by "nfo:fileSize >= 2000000". This also works for nested queries. A pattern like "related to %% ," matches "related to" followed by any number of terms, then either the end of the string or a comma. The terms matched by %% are all fed to the C++ rule implementing the pattern, that can concatenate them in a AND and use them as a simple value for a relatedTo comparison.

  When every possible pattern has been tried, the input is considered to be parsed and is returned to the caller. The parser did its job.

  What is a bit special with this parser is that matching patterns looks like matching regular expressions. Sure, patterns are more specific than regular expressions (and to be complete, regular expressions can be used in patterns, so that the translators do not need to explicitly write the dozens of forms a word can take in some languages), but it's the same spirit. This parser, even though not based on regular expressions anymore, and more robust, is not following a precisely defined syntax. It's up to the translators to write rules (the syntax is very simple), and each language can have its own "syntax". I find that way more user-friendly than a syntax fixed in stone.

  The Query Builder Widget

  The second part of the project was to implement a query builder widget. This widget is responsible for helping the user to enter queries. The first help is syntax-highlighting: the widget highlights what is recognized by the parser, so that the user can discover that things like "modified last week" (yes, I ported the human date-time parser to the new infrastructure) is understood by the parser and can be used when needed. The second help is auto-completion: the widget shows what patterns correspond to what has been written. For instance, entering "tagged" yields "tagged as ".

  There is no complex (or boring, depending on your point of view) algorithm to explain in this part. The code is simple, and what took time was to design a user-friendly widget that is actually useful without getting in the way of the user. A picture is worth a thousand words:

  

  And in French:

  

  A Firefox and Konqueror Plugin Indexing User Emails

  At the middle of August, most of the first two parts of my GSoC project were nearly completed. The rest of the coding period was dedicated to bug fixes (there were many small and difficult-to-fix bugs that only happened on select queries). I also added unit tests for the parser, and I submitted a patch to Dolphin to use the query builder widget in it (see the screen-shot above).

  The problem is that it is impossible to write unit tests during a full month, especially when the project tested was developed during two months. I therefore decided to find something I could do and that would benefit to Nepomuk.

  I stared with the observation that many users, even my Computer Science class mates, use webmails. Nearly nobody use desktop mail clients anymore, and it is a shame given how useful they are. Moreover, people that use mail clients, even on Linux, don't all use KMail (there are GNOME users or KDE users that are used to Mutt or Thunderbird, or that simply don't want to use KMail). This observation made me think that only a small portion of the KDE users have their mails indexed in Nepomuk (by Akonadi), and therefore that email-related queries enabled by the new query parser are of no use to the others.

  The goal of my project was to seamlessly allow these users to enjoy the features of Nepomuk. I did not want to oblige them to enter personal identification data (like POP/IMAP account names, servers and passwords) every time they open their KDE session or at the installation of their Linux distribution. Everything had to be completely transparent, without the need of any password.

  The only solution I have found is to develop a browser plugin that monitors the activity of the user and recognizes when he or she is visiting a webmail. When it is the case, a webmail-dependent Javascript snippet is run to extract the mails displayed on the page. Currently supported webmails are Yahoo! Mail (in French), Roundcube and Outlook. GMail can also be parsed but its insane use of AJAX and asynchronous queries make it difficult to detect when the page has changed, and when the parser must be re-run on it.

  A nice side-effect of this project is that Nepomuk is now able to index e-mails handled by Thunderbird or any other mail client, because I implemented a MIME/mbox file indexer.

  More information about this project, that is still in an experimental state, can be found in this blog post.

  Future Work

  I now have a KDE Developer account, and I want to use it to improve KDE. My studies require a fair amount of work, but I will try to work on two projects that interest me:

  • As suggested by Vishesh Handa, Nepomuk needs a good MS Office file indexer (able to parse the old binary formats still widely used). I have seen in the Calligra source code that a library able to parse these files is available, and I will try to use it (at least to extract the plain text content of the files).
  • The pattern matcher I use in the query parser could be useful to other projects. I will see if it is possible to split it out of Nepomuk (or to develop a standalone version) and to make it available as a shared library, that could be used by other applications. The date-time and miscellaneous parsing passes (like the recognition of numbers and file sizes) will be available in this library, as they can be useful. I hope that people will be able to develop interesting applications with this parser.

  So, stay tuned, and you can monitor my blog for activity on other projects (Planet KDE only shows blog posts related to KDE).

• A Nepomuk Integration Plugin for Firefox

  Published on ven 06 septembre 2013 in Nepomuk, (Comments)

  Since the beginning of August, I'm experimenting with the idea of integrating Nepomuk and the most used web browsers. I started with a Konqueror plugin that detects when the user is reading an e-mail on a webmail in order to index the e-mail in Nepomuk. That was quite a challenge, but I'm now able to detect e-mails on Outlook, Yahoo! Mail, any Roundcube-based webmail, and GMail will come shortly (this webmail is very difficult to parse).

  I chose Konqueror because I largely prefer to develop extensions in C++ (a compiled language with clear APIs, and the wonderful auto-completion of KDevelop), and I didn't want to fight with webmails and with Javascript at the same time.

  The Firefox Extension

  This week, I ported the Konqueror extension to Firefox. You can find it in my scratch repository. The extension can be found in the firefox/ directory and is currently unpackaged. If you want to get a proper .xpi file, just do the following:

  1 2	cd firefox/ zip -r ../nepomukintegration.xpi *

  After having installed this extension, you can begin to visit webmails. Outlook currently works best (I don't know how it is possible, but the HTML code is very well designed and crystal clear). You need to open e-mails to have them indexed, as the extension has no way to know the content of emails if you don't open them.

  When you have read some emails, go to /tmp/kde-YOU/ and look for .mime files. If you open them (using KWrite for instance, the MIME format is very human-readable), you can see the content of your e-mails. There will be no "Date" field as I haven't yet figured how to implement this C++ function in Javascript:

  1 2 3 4 5 6

  QDateTime stringToDate(const QString &string, const QString &format, const QString &language) { QLocale locale(language.toString()); return locale.toDateTime(string, format); }

  You can also download files using Firefox. Every download is represented by a .webaction file in your temporary directory. This file starts with "DOWNLOAD" (more actions will be possible in the future, like "BOOKMARK" or "HISTORY"). The next lines are the website on which the file has been downloaded, the URL of the file, and the path where you downloaded the file.

  Integration with Nepomuk

  Now that your temporary directory is full of files that describe your web activity, you can pull the steckdenis-webindexers branch of my Nepomuk-Core repository. Build this Nepomuk-Core and install it where you have your KDE developer installation. Now, go to your temporary directory and type something like that:

  1 2	nepomukindexer 8767d79459fbc4720083d06bed365236.webaction nepomukindexer 88d86afe9938e87aa3324e3cd6ff8a55.mime

  These two operations index the data that Firefox have just extracted in Nepomuk. You can also ask Nepomuk to watch your temporary directory and to index files in it automatically when they appear (this can be done in the "Desktop Search" configuration module). The result is as shown below:

  

  Conclusion

  First, I want to precise that the indexing of downloads is based on work by Martin (for the Firefox part) and Vishesh (for the Nepomuk part). I'm only responsible for gluing the Firefox part in my Firefox add-on and for adapting the Nepomuk part as a Nepomuk File Indexer.

  This ends my experiment of indexing e-mails in Nepomuk. This experiment started because I thought that it would be good to allow users that use webmails to enjoy the niceties of Nepomuk. Not everyone uses KMail, that already index everything in Nepomuk.

  If I remember correctly, the GSoC ends in two weeks. What I will try these coming two weeks is to merge my work in Nepomuk (the query parser and the query builder widget) and in Dolphin (I have a small patch that makes Dolphin use the new query builder and that I would like to merge if a future Dolphin version is planned). I will also propose this experiment for merging. The Nepomuk part is not invasive at all (I just added three file indexers : MIME/MBOX, vCard and web actions), and the Firefox add-on just have to be uploaded to Mozilla Add-Ons.

• Finishing Touches and GMail

  Published on jeu 29 août 2013 in Nepomuk, (Comments)

  It has been a long time since my last blog post, but that doesn't mean nothing new has happened. In addition to fine-tuning the new Nepomuk Query Parser, I also try to write a Konqueror plugin that detects emails on common webmails and that index them in Nepomuk.

  Fine-Tuning of the Query Parser

  At the beginning of August, the query parser was already in quite a good shape. It was able to parse many queries in English, and I translated it into French the check that everything goes well. The first thing I discovered is that even if the translation operation was easy, it could be even better.

  The problem was that the pattern matcher syntax was not adapted to languages with rich "declination". In English, adjectives are always spelled the same, independently of the words around them. In French, adjectives can have four forms, as they have a genre (English people always find funny that "A car is a lady in French"), and a number (the singular form is different from the plural one).

  An example shows the problem better: consider the "modified|changed at %1" pattern, that matches "modified at 3.20pm" for instance. In English, two different adjectives are given, and the pattern works well for "files modified at ..." and "file modified at ...". In French, the pattern needs to accommodate the four forms that each adjective can take. It becomes "modifié|modifiée|modifiés|modifiées|changé|changée|changés|changées à %1". It's quite long, isn't it ?

  The first step towards a solution is to allow regular expressions in patterns. The pattern in French now becomes "modifiée?s?|changée?s? à %1", its length is again under control. Regular expressions is the best I can give to translators, even if I think that many languages are still too complex to have each word expressed using a single expression. That is not a problem as it is still possible to use several expressions in a single pattern, so at worst the translators just need to write the 60 forms of their words, separated by pipes.

  

  The problem of this solution is that the pattern is shown to the user, as you can see on the above image. Patterns that don't use regular expressions are nice to display (they are words separated by pipes), but regular expressions are way more difficult to understand. Now that patterns are shorter but not user-friendly anymore, I needed to find a way not to have them displayed.

  This is done in two steps. The first one is that the user input is displayed when possible. If the user has typed "Modified at", the auto-completion box will show "Modified at", not "modified|created at|on". This already simplifies the auto-completion box, and has the nice property of respecting the case and the exact spelling the user used.

  The second step is to display only one possible literal per term after the user query. If the pattern is "not touched|modified|changed since|after %1" and the user entered "not modified", the completion proposal will be "not modified since [something]". Translators can now hide the regular expressions: they just have to start their list of regular expressions with a simple nice word, that serves as a "title": "modifiée?s?|changée?s? à %1" becomes "modifié|modifiée?s?|changée?s? à %1", and the auto-completion popup will only display "modifié", never the following regular expressions.

  Extracting Information from Web Pages using Javascript

  By "parsing", I mean "recognizing the content of specific elements of a page". At the beginning of the month, I started to experiment with extracting emails from webmails and indexing them in Nepomuk. The first extractor I made is the one for Yahoo! Mail Neo (the French version). This extractor works very well.

  After that, I wanted to quickly implement a parser for GMail and Outlook, two of the most used webmails. I started with GMail, and discovered that the current infrastructure did not allow to parse it. The problem is that this webmail is very complex. For instance, an email can be displayed on its own page (new messages from unknown contacts) or in a discussion (when you reply to someone). The discussion is especially difficult to parse as information about the emails is scattered on the whole page: the content of the messages is in the discussion itself, but the meta-data (date, sender, title) is contained in elements very far from the messages in the DOM tree.

  I therefore decided to replace my own JSON-based syntax with full-fledged Javascript. QtScript made that very easy, and in one day, I had the Yahoo! Mail parser rewritten in Javascript:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44

  ({ urls: ["*.mail.yahoo.com/neo/*"], emails: function(document) { var content_nodes = document.findAll("div.message.content"); var mails = new Array(); var field = function(parent, name, attribute) { var nodes = parent.findAll("div.info.expanded ul[aria-labelledby=\"hdr-" + name + "\"] a"); if (attribute == "") { return contentOf(nodes); } else { return attributeOf(nodes, attribute); } } for (var i=0; i<content_nodes.length; ++i) { var content_node = content_nodes[i]; var date = content_node.findFirst("div.info div.date"); var title = content_node.findFirst("div.info h3 span.txt"); var content = content_node.findFirst("div.msg-body"); if (date == null || title == null || content == null) { continue; } mails[i] = { title: title.content(), content: content.content(), date: stringToDate(date.content(), "ddd, d MMM. yyyy à h:mm", "fr"), from_name: field(content_node, "from", ""), from_addr: field(content_node, "from", "title"), to_name: field(content_node, "to", ""), to_addr: field(content_node, "to", "title"), cc_name: field(content_node, "cc", ""), cc_addr: field(content_node, "cc", "title") } } return mails; } })

  The JS rules can use everything exposed by QtScript (I think that it is standard core EcmaScript), plus some helper functions implemented in C++. The DOM Elements are represented by JSElement objects (each JSElement being either a JSElementKHTML or a JSElementWebkit), and they have a small set of methods:

  • tagName() that returns the tag name of the element
  • content() that returns the text content of the element
  • attribute(name) to get the value of an attribute
  • findAll(selector) that returns the list of all child elements matching a CSS rule
  • findFirst(selector) that returns the first element returned by findAll, or null if the list of matched elements is empty

  contentOf takes a list of elements and returns a list containing their content, and attributeOf does the same for an attribute. Finally, stringToDate(string, format, lang) converts a string to a date using a specified locale. Internally, QLocale::toDateTime is used.

  Detecting Changes in the DOM Tree of a Document

  After having implemented rules written in Javascript, I decided to try again to parse GMail. Its rule is a bit more complex than the one of Yahoo, but is was possible to implement.

  The problem is that when I tried to test my rule, nothing happened. Why ? Because the Konqueror plugin needs to detect when the DOM tree of a page changes. This notification triggers a re-parse of the page, and the parsing rules can detect potential new emails. In Yahoo! Mail, the DOM changes are correctly detected: they are preceded by a network request that can be detected by using the KParts::ReadOnlyPart::started(KJob *) signal. GMail does not trigger this signal, though. By sniffing my network, I discovered that GMail also performs network queries just before updating its DOM tree, but these queries are not detected. I don't know why and I have asked on the kde-devel mailing-list. If you have a solution for reliably detecting DOM tree changes, you will have my eternal gratitude.

  I will try to solve this "GMail problem" in the coming days. After that, I will implement rules for Outlook and Roundcube. This will allow me to check that the current Javascript-based infrastructure is right, and that I can proceed to the development of a Firefox plugin. If you have ideas regarding the Nepomuk Query Parser or find bugs in it, don't hesitate to tell me as I very muck like to work on it, and I want to make it the best possible.

• A Nepomuk Integration Plugin for Konqueror

  Published on mer 07 août 2013 in Nepomuk, (Comments)

  Last week, I presented an idea about indexing messages from webmails in Nepomuk. The summary of this idea is to implement a browser extension for Firefox, Chrome and Konqueror. This extension parses the DOM tree of every page visited by the user that belongs to a webmail. When e-mails are found, they are extracted and stored in a temporary file. Nepomukfileindexer then quicks in and indexes these e-mails in the Nepomuk database.

  The reason why I started this experiment is because many users use webmails instead of mail client programs like KMail or Thunderbird. The information about these emails cannot therefore be indexed by Nepomuk. As there are many different webmails, the biggest part of the project is to implement parsing rules for each of them. The browser plugin itself is quite small.

  Extracting Information From the DOM Tree

  Last week, I implemented the file indexers that are responsible of indexing mbox files and MIME files into the Nepomuk database. These indexers are now in a fairly good shape. This week, I started to implement the browser plugins. I wanted to start with Firefox, as it is the most used web browser, but my experience with Javascript is quite limited. I therefore started with Konqueror, more on that later.

  The browser plugin reacts on DOM tree changes. When the DOM tree of a page has changed, the URL of the page is used to determine on what webmail the user is. If rules exist for this webmail, they are applied.

  A rule is a simple JSON file that describes how to extract information from the DOM tree. I use JSON because parsing it in C++ is easily done using QJSON, and parsing JSON in Javascript is as easy as an eval call. The DOM is read using CSS selectors, that is to say selectors as used in CSS style sheets (div.test matches every <div> tag that has the class "test"). The first rule I developed is the one for Yahoo! Mail in French (Yahoo! Mail has the bad idea of displaying a strongly localized date-time, and the ISO-formatted date-time is found nowhere in the document):

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17

  { "urls": ["*fr*.mail.yahoo.com/neo/*"], "date_format": "ddd, d MMM. yyyy à h:mm", "emails": { "node": "div.message.content", "date": { "node": "div.info div.date" }, "title": { "node": "div.info h3 span.txt" }, ... "content": { "node": "div.msg-body" } } }

  Applying this JSON pattern against the DOM tree yields a tree of QVariant values. Each QVariant is either a list, a map or a string. The tree looks like that:

  1 2 3 4 5 6

  emails = [ {"date": "Mar, 7 aoû. 2013 à 13:12, "title": "This is a mail", "content": "Hi\n\nThis is an email!\n\nCheers" } ]

  This data is stored into a MIME file using the wonderful KMIME::Message class. The MIME file is stored in /tmp/kde-username/.

  The Konqueror plugin

  • The source code is available my nepomukintegration scratch repository.

  The Konqueror plugin is currently only 300-lines long. QJSON does all the JSON parsing, KMIME::Message writes the MIME files. What is left to do in C++ is matching the patterns against the DOM tree. Here, Konqueror acts as two web browsers: you can choose to display web pages using either KHTML or kdewebkit. The two engines do not share the same structure. KHTML is based on KDE specific DOM classes, and kdewebkit uses a QWebView.

  Matching a pattern therefore needs to be done for the two engines. Instead of duplicating all the code, I used a big template function. When the APIs of the two engines are different, I implement utility functions with an overload for each engine:

  1 2 3 4 5 6 7 8 9

  static QString elementContent(const DOM::Element &element) { return element.textContent().string(); } static QString elementContent(const QWebElement &element) { return element.toPlainText(); }

  The template functions only uses the utility functions, and the compiler chooses the right ones.

  Status

  The Konqueror plugin is now quite usable. It works, it does not crash, and my emails are found. In fact, my emails get properly extracted to temporary files, and running nepomukindexer correctly indexes the data in my Nepomuk database. The plugin can therefore be considered as starting to be functional.

  Indexing mails is one thing, contacts is another one. "Simple" contacts are already indexed: if the sender of an e-mail (identified by its name and its e-mail address) is not already present in the Nepomuk database, a new contact is added. By parsing contact-specific pages of webmails, it should be possible to extract more information about the contacts, like their birthday, gender, etc. This information will be stored in vCard files, indexed by the vCard file indexer I developed last Sunday. Another data I would like to index is history.

  You may be wondering why I'm storing the extracted information in files instead of storing it directly in Nepomuk. In fact, Konqueror is a KDE application and has access to Nepomuk. The reason is that I don't want to duplicate work. Nepomukindexer already handles the cleanup of existing meta-data, and my email indexer ensures that an email never gets indexed more than one time. Moreover, the extension needs to be quite "cross-browser compatible", and Firefox and Chrome have no access to Nepomuk (Nepomuk requires a Qt event loop, and integrating the Qt event loop in GTK-based applications is tricky, see what the GTK Qt style has to do).

• Extracting Information from the Web

  Published on ven 02 août 2013 in Nepomuk, (Comments)

  Google has just sent me a mail saying that I passed the midterm evaluations of the Google Summer of Code! Yay! I want to thank everybody that helps me in my GSoC project, everyone on the Nepomuk mailing list, and more generally everybody part of the KDE community. Everyone is very friendly, and working on KDE is a very enjoying experience!

  As the query parser and the query builder widget come close to being usable (I have still bugs to fix, and things to change in order to make the query parser more localization-friendly), I'm beginning to think about other things on which I can work. One of these thinks is storing in Nepomuk information that comes from the user's web browsing.

  The Problem

  Nepomuk works really well, and now has a good query parser and a fairly nice query builder widget (I think). You can index your files, meta-data about them is extracted and stored in the database, and you can query this database using Dolphin or KRunner. The problem is that this indexing is currently limited to files.

  If you use KMail or any Akonadi-based application, your e-mails and contacts are also stored in Nepomuk, ready to be queried using queries like "mails from John". The problem is that KMail is not used by every KDE user, as some of them prefer other mail clients (that don't use Akonadi). In fact, the majority of the KDE users use webmails, and their instant messaging applications are not KDE ones. Furthermore, many Linux distributions install Firefox as the default web browser. Most of our users therefore don't enjoy the Nepomuk integration that KMail, Kopete and other applications like that provide.

  Indexing Information from the Web

  My idea is to index in Nepomuk information found on the web, but not automatically, not by using a web crawler. The user itself will provide this information by browsing the web. The idea is to implement a Firefox add-on (and after that a Chrome add-on, as Chrome is also widely used and it seems that Rekonq supports or will support Chrome add-ons) that extracts information like that (if the user allows it to do so) :

  • The fact that a page has been visited (I just have to check that Nepomuk has an ontology for storing web history)
  • If the page is recognized as being one of a known web-mail, its DOM structure is explored to find information about a displayed mail or contact
  • When downloading files, the URL from which it is downloaded can be stored in Nepomuk (there is already a Firefox add-on that does that)

  More File Indexers

  As Vishesh Handa once explained in a blog post, Nepomuk uses file indexers to extract information about files (line count, video format, information about photographs, etc). Adding a file indexer is very easy, even complex ones are only a couple hundred lines long. The file indexer gets a file URL, passes it to a library (ffmpeg, taglib, etc), gets information and stores it into Nepomuk.

  For my new experiment of storing web information in Nepomuk, I have to develop three new file indexers:

  • An e-mail indexer. It uses the wonderful KMIME and KMBOX libraries (part of kdepimlibs) to open mbox files and to index the mails it contains. For maildir directories, the e-mails are directly indexed using only KMIME. The indexing consists of storing in Nepomuk the plain text content of the mails and their headers (sender, receiver, etc). Senders and receivers are also used to set the e-mail address of contacts. This is interesting because the office extractors (MS Office and ODF) already create contacts for the authors of your documents. This indexer can therefore add an e-mail address information to these contacts.
  • A vCard indexer, that uses the KABC library of kdepimlibs to extract detailed contact information from vCard files (name, street address, phone number, etc).
  • An indexer for nepomuk-specific file formats. For instance, a file format can represent associations between downloaded files and their original URL. Another one can represent a piece of navigation history.

  Two of these indexers use the wonderful libraries developed by the kdepimlibs developers. They are rock-solid and implement very complex standards (MIME files are hairy to handle). I want to thank them very much for these libraries that are a pleasure to use:

  1 2 3 4 5 6 7 8 9

  KMime::Message message; message.setContent(raw_data_from_file); message.parse(); resource.addProperty( NMO::plainTextMessageContent(), message.decodedText(true, true) );

  Bridging the Web and Nepomuk

  Why do I use file indexers to index information coming from the web ? Because browser extensions are limited. Firefox and Chrome are both based on GTK+, and integrating Nepomuk and its Qt event loop would be terribly difficult. Moreover, Chrome does not allow native extensions at all (I'm not sure about that, my information comes from very early Chrome versions).

  Browser extensions can perform HTTP queries, though, but I do not want to develop an HTTP server that would listen to POST request to create Nepomuk resources. I have already developed an HTTP/1.1 parser (with support of chunked encoding etc), but it is buggy and fairly complex.

  The solution lives in files: Firefox and Chrome extensions both are allowed to create local files (it's easy to do in Firefox, less so in Chrome, but possible). So, when a piece of information is extracted from a web page, the extension only needs to format it according to a web standard (MIME, mbox, vCard, etc) or a Nepomuk-specific one (for history and downloaded files), and then to store it in a temporary directory monitored by nepomukfilewatch. nepomukfilewatch is a service that indexes files when they are created or modified, so it is possible to detect these new files and to index them. The bridge is in place.

  Using file indexers also has a nice side effect: users using Thunderbird/Mutt/etc can point nepomukfilewatch to their e-mail directories (it is a simple as checking them in the KCM module of Nepomuk), and have their e-mails and contacts indexed by the e-mail indexer. Akonadi users don't have to do anything as Akonadi already stores information in Nepomuk. And as the e-mail indexer skips e-mails already stored in Nepomuk (a mbox file only grows and can contain thousands of mails, it would be way too slow to re-index them at every run), if a mail folder is indexed by both the e-mail indexer and Akonadi, there will be no conflict.

  Current status

  As this projects has not (yet) been approved by the Nepomuk developers, all the code still lives in the steckdenis-webindexers branch of my nepomuk-core git repository. The code is still minimal, but the e-mail indexer is already there. It can index mbox files and plain e-mails stored in maildir directories. I plan to implement a vCard indexer shortly.

  As this project is in a very early stage, I encourage you to comment on it and to give me your impressions. Do you think that this architecture is right ? Is my use-case a good one (the fact that most users nowadays use webmails) ?

« Previous Page 5 / 11 Next »