Articles in the Nepomuk category

• The DOM and your wish-list

  Published on mar 05 août 2014 in Nepomuk, (Comments)

  This week is the last of the "active coding" period of the Google Summer of Code. On Monday 11, students should stop implementing features and start documenting things, fixing the remaining bugs and ensure that everything is working.

  I don't have a huge list of new features to present this week because I had to fight a very nasty bug (and it is still not entirely resolved), but here is nevertheless an addition to the QML/Javascript language support plugin for KDevelop that may be quite useful to a lot of people: support for the Javascript DOM.

  The Document Object Model

  Last week, I added support for Node.js modules. I did that because several users of the QML/JS plugin told me that they regularly work on Node.js code. This week, despite a bug that makes the unit tests fail (but not the real-world KDevelop), I decided to add support for the Document Object Model of Javascript, so that web developers have their scripts fully supported by KDevelop. As libraries like jQuery also rely on the DOM, having it supported is a good thing.

  The DOM is really huge and contains hundreds (if not thousands) of classes. Each HTML element or CSS property has its own class, and there are even classes for things like cookie handling, message passing between web pages, XmlHttpRequest, etc.

  I was not very fond of the idea of writing Javascript description files for all these classes. Luckily, I received an unexpected help here: Webkit (the web engine) also has to support these classes, and instead of implementing them directly in Javascript on C++, it ships files having a .idl extension. These files describe the API of the different classes and look like this:

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15

  module window { interface [ ...ignored... ] DOMWindow { attribute [Replaceable] BarInfo locationbar; attribute [Replaceable] BarInfo menubar; attribute [Replaceable] BarInfo personalbar; attribute [Replaceable] BarInfo scrollbars; attribute [Replaceable] BarInfo statusbar; DOMSelection getSelection(); [Custom] DOMWindow open(in DOMString url, in DOMString name, in [Optional] DOMString options); } }

  The statements between brackets are hints for the Javascript binding generator and can be ignored. What is interesting in those files is that the attributes, methods and arguments are all named and typed. This means that no information is missing! I just had to write a Python script that parses one of these files and produces a .js equivalent, and I was able to add support for all the DOM.

  This morning, I've pushed a big commit to the kde:kdev-qmljs repository. It adds more than 10000 lines of .idl and .js files (I ship the IDL files so that looking at them and possibly modifying them is easier, but their real origin is Webkit). The Javascript file describing the DOM is approximately 4000 lines long and takes a bit more than two seconds to parse. In order to keep it to a reasonable size, I skipped the Canvas classes, the HTML classes (having Element is enough, no need to have HTMLLink, HTMLParagraph, HTMLSpan, etc), the File classes (web storage), the XML parser classes and several other ones. They are rarely used and would have made the description file far too big and too slow to parse. I can still add these classes if many users need them, though.

  Here are now some screenshots:

  

  Creating a DOM element.

  

  Code-completion for DOM classes.

  

  Code-completion for global variables and functions (that are in fact members of "window", yes, "window.alert()" works in JS).

  Finishing touches and wish-lists

  The DOM was the last big feature planned for the QML/JS plugin. In fact, I'm very happy to have been able to go this far in the project. My original GSoC proposal was entirely oriented towards QML, with only basic support for Javascript (just enough for Javascript snippets to be usable in QML files).

  I've still some bugs to fight and some things to improve. For instance, having "class HTMLElement inherit Element" displayed in the navigation widget instead of "function HTMLElement()" could be a nice addition. Allowing the user to add include paths (for Node.js and QML modules) could also be great, because some QML modules are not installed at the same place as the others.

  But what I would really like to do is to implement what daily QML developers may want. I have posted a message on the plasma-devel mailing list asking which features Plasma developers (regular QML users) may want. I've already got some hints, but don't hesitate to tell me what you would like to have implemented in kdev-qmljs. I will also be in Randa, and while most of my time will be dedicated to porting the QML/JS plugin to KF5, I will also try to listen to the wishes of everyone present.

• Node.js support in KDevelop

  Published on ven 25 juillet 2014 in Nepomuk, (Comments)

  This is a quick blog post to say that I'm working on support for the Node.js framework by the KDevelop QML/JS language plugin. The first part of my work is already finished: "require" is properly recognized and Node.js modules are properly looked-up and included. The "exports" and "module.exports" identifiers are also recognized and allow Javascript files to export symbols and to be used as modules. This means that you can now edit Javascript files using Node.js and have most of the modules recognized, included and parsed.

  The second part is writing huge Javascript files for the built-in Node.js modules. "http", "buffer", "fs", "url" and many other modules exist only as part of Node.js. There are Javascript files for them in the Node.js Git repository, but these files don't allow the QML/JS plugin to infer the types of the methods. They are also fairly big and would need a big cleanup in order to be fast enough to parse.

  As with ECMAScript, I solve the problem by shipping with QML/JS stripped down versions of those files. These files describe the classes, methods and attributes of each module, and follow a very simple format. This code is not designed to be run, only to be parsed by the QML/JS plugin, that will discover the types of the function parameters and their return type.

  1 2	exports.spawn = function (command, args, options) { return new exports.ChildProcess(); }; exports.spawn("", [], new Object());

  Writing all these files is tiring and will take some time (in fact, I'm writing them for a bit more than one day, now). So, there may be fewer commit than usual in the QML/JS repository, but I hope that this feature will serve many users. Here is a teaser of what is coming (note that functions are now listed in a way more human-friendly form):

  

  Once this feature is finished, I will concentrate my efforts on the QML code-completion. I'll also experiment with the KDE Frameworks 5 (and the KF5 version of KDevelop), because I'd like to offer a KF5 version of the QML/JS plugin. The reason is twofold: Qt5 offers the wonderful QtQuick.Controls module (it will allow me to implement many navigation widget, that will assist the users in choosing a font family, configuring and previewing animations, etc), and early KF5/Plasma5 adopters may be well pleased to be able to use the latest and greatest KDevelop on their latest and greatest environment, while still being able to work on QML and Javascript files.

  UPDATE: Here is an additional screenshot that shows that child_process.ChildProcess is correctly recognized, and that it correctly inherits from event.EventEmitter (you see that addListener is in the list), which in turn inherits from Object (there is an "assign" method at the very bottom of the list). Having all of this working was a bit more difficult than expected, but now function prototypes and class inheritance seem to work.

  

• QML binary modules

  Published on dim 20 juillet 2014 in Nepomuk, (Comments)

  Developing a new project is usually done in two phases: first, the most important features are added, and the feature list of the project quickly grows. Then, the features need to be polished, bugs fixed, and the overall usability of the project improves slowly. This second phase is often considered less interesting than the first one, but personally, I like to fix small annoyances. Fixing them makes the difference between an experimental project and one that can be reliably used by real users.

  During this GSoC, I try to work in small steps, implementing a big feature and then improving it, then implementing the next feature, and so on and so forth. This works quite well and allows me to keep a high motivation in what I'm doing: implementing a big feature is exciting but also exhausting, and improving existing features mainly consists of many small and easy commits that each have a noticeable effect. This week, I'll talk about one major feature and several small corrections that I've implemented for the KDevelop QML/JS language plugin.

  The fixes

  Javascript has no syntax to express that "this function is a member of this class". It is not directly possible to use something like function Class.method() as in C++. Developers are therefore obliged to declare class methods using function expressions:

  1	Class.prototype.method = function (a, b) { return a + b; };

  The fact that method is assigned a function has an annoying implication for the QML/JS plugin: when method is declared, it is a simple variable, a member of Class that can contain anything (a function, an integer, etc). This means that method is declared as a Declaration, not a ClassFunctionDeclaration. This is not really a problem, but this prevents KDevelop from displaying the right icon alongside the methods:

  

  The solution is quite simple: when a Declaration has a function type (when it is actually a function), then the QML/JS plugin ensures that a function icon is displayed. An icon may seem unimportant, but such a bug is very easy to fix (one or two lines), and not fixing it would have immensely annoyed all the users of the plugin.

  

  The other small feature that I've added is support for the __proto__ member of Javascript object instances. This member is used by several Javascript libraries and may also be used in a couple of QML files, so it is not going to be useless:

  

  Other small bugs have been fixed. For instance, in a QML file, typing ANamespace. lists the components declared in the namespace, and nothing else (previously, there was some garbage, for instance the namespaces imported by ANamespace, or the built-in Javascript classes). Code-completion for import statements is now way more useful, but more on that later.

  Binary QML module imports

  Back to the QML world. The goal of my GSoC is to have the best possible QML support in KDevelop, because KDevelop is already the best-in-class IDE for C++ Qt development, and only lacks a good QML support. Javascript, on the other hand, already has many great IDEs and competing with them may be a bit out of the scope of a GSoC project.

  My strategy to have great QML support in KDevelop is to concentrate most of my efforts on this language, while still having a robust enough Javascript support so that Javascript snippets in QML files work as expected. For instance, Javascript built-in classes were added because many of them are used in QML files.

  So, this week was dedicated to importing QML binary modules and QML installed modules. First, some definitions:

  • QML installed module: QML module that is installed on the user's system, for instance by a KDE package. Unfortunately, QML does not have system-wide include paths as Python and Ruby do, each QML application provides its own search path. Luckily, most QML applications either use only Qt-provided modules (and they have been supported for a long time now), or use KDE-provided ones. KDE installs all its modules in /usr/lib(64)/kde4/imports, so they are quite easy to locate. An installed module consists of a directory that contains .qml files and possibly .so files. The library files (that can be mixed with plain .qml files) contain additional native components, implemented in C++.
  • QML binary module: Library that contains QML components. The .so files I was talking about hereabove are QML binary modules.

  Supporting QML installed modules was quite easy: KDE's KStandardDirs allows me to easily locate them, and the infrastructure for importing whole directories was already in place.

  Binary QML modules were a bit more difficult because they need to be dumped before the QML/JS plugin can use them. Dumping a binary module file is the action of running qmlplugindump on them, a tool provided by Qt that takes a .so file and writes a QML module file for it. The QML module file is a valid QML file and can be parsed by the plugin.

  With directory imports working, binary QML modules dumped and the dumps correctly parsed, the all the pieces fit together:

  Uh oh! A .so file!

  Enter qmlplugindump (the file is cached for later reuse)

  Components exported by Plasma Core

  Members of a binary QML component

  Finally, as an added bonus, I've improved the code-completion for import statements. They now list only the imports that start with what the user has already typed, and they handle installed modules:

  

  Testing the QML/JS plugin on real QML files has shown that QML is very nicely handled, even if there are still some small glitches (for instance, several built-in QtQuick 1.0 types and properties are still missing). Sven Brauch will also send me a list of places where the code-completion could be more useful. In fact, don't hesitate to tell me which features you would like to see implemented, I've still four weeks ahead of me!

• ECMAScript 6 support in KDevelop

  Published on mar 15 juillet 2014 in Nepomuk, (Comments)

  Several weeks ago, KDevelop became able to display the list of all the QML components shipped with Qt, along with their properties, signals and methods. This week, the same happened for Javascript: all the standard ECMAScript 6 classes and objects are now recognized by the KDevelop QML/JS language plugin.

  Adding this support was easy but not the funniest part of the project :-) . I had to read the ECMAScript specification (more specifically the sections about the built-in objects) and to write a big Javascript files that describes the built-in classes and objects. This file declares all the classes and all their methods and attributes. Assignations and calls are also performed so that the QML/JS plugin can infer the type of the attributes and the arguments of the methods. The file is 1133 lines long and looks like this:

  1 2 3 4 5 6 7

  Number.prototype.isNaN = function (number) { return true; }; Number.prototype.isNaN(1.0); Number.prototype.isSafeInteger = function (number) { return true; }; Number.prototype.isSafeInteger(1); Number.prototype.MAX_SAFE_INTEGER = 1;

  Here, an isNaN method is declared. It contains a return statement so that the plugin knows that this method returns a boolean. There is also a call that allows the plugin to infer that the argument is of type float. The second methods is declared the same way. It also returns a boolean but it takes an int as argument. Finally, an attribute is declared.

  Note that the ECMAScript specification makes the distinction between the methods/attributes of the Number class (that should be declared using Number.MAX_SAGE_INTEGER=1; I think) and the methods/attributes of Number instances (declared using the prototype). As the QML/JS plugin currently does not differentiates class members and prototype members, all the declarations are currently visible both in Number and in Number instances.

  The screenshots

  Now that the QML/JS has a big list of declarations, they can be used. The simplest use of the declarations is simply to show them to the user:

  

  Math and JSON are interesting objects because they are also widely used in QML files. In fact, I decided to add declarations for built-in ECMAScript types because I discovered that they were so heavily used in QML.

  The second (and most interesting) feature is to provide code-completion items for variables and members having a built-in Javascript type like int, float, string or array:

  

  

  Support for RegExp literals

  Finally, adding support for built-in ECMAScript classes allowed me to support RegExp literals. They were not supported before because KDevelop does not have a "regexp" type, so the language plugin has to provide one itself. This is now done, and regular expressions are now fully supported:

  

  

  The last screenshot might be quite pleasing to RegExp users: the return value of RegExp.match is known to the plugin, which can therefore display useful code-completion items. How this special return value is supported is a bit dirty and involves code like this (and a good type inference engine):

  1 2	RegExp.prototype.match = function (string) { return {index: 1, input: "", length: 1}; }; RegExp.prototype.match("");

  The match function returns an object having three special keys, that become available to any code calling RegExp.match.

  Now I have to add support for the DOM, and I think that this task may be even bigger than ECMAScript. The DOM is so huge, but supporting all of it is a prerequisite to supporting big Javascript libraries like jQuery. I'll also look at Node.js, which seems to be used by several early users of the plugin. Maybe having basic support for Node.js (and its require function) may be possible without having to describe the whole DOM.

• Support for Javascript prototypes and QML directory imports

  Published on jeu 10 juillet 2014 in Nepomuk, (Comments)

  Here is a new status report of my work on the KDevelop QML/JS language support plugin. This week, I have fixed several small bugs and implemented two features: support for function prototypes (a Javascript-only feature), and support for directory imports in QML.

  Function prototypes

  I think that this was the feature the most difficult to implement of all the GSoC project. Moreover, support for function prototypes and Javascript object is not yet complete, and I doubt that the plugin will be able to support the whole Javascript prototype-oriented programming. The reason is that Javascript is a very dynamic language while KDevelop is very static. I'll show you examples of code snippets that would require, in fact, a complete Javascript interpreter and type tracer for them to be correctly handled in KDevelop.

  Currently, the feature starts to be usable but is still minimal, mainly because my knowledge of Javascript prototypes is also minimal. I've implement only what I've understood and what was possible without adding thousands of lines of code to the QML/JS plugin. The goal of my GSoC is to have the best QML support possible, with decent Javascript support.

  The main idea of the prototype support is to associate to every function a "prototype". Here, the word does not have the exact same meaning as in the Javascript literature. My definition of the prototype of a function is "the context of the this identifier in this function". Several functions, if part of the same class, can therefore share the same prototype, because "this" will have the same value in all of them. Moreover, there is no distinction between the prototype of a function and the objects created when calling new Function().

  

  In the above screenshot, a function SayHello is declared. As it is a function declaration (with the function keyword), a new prototype is created and assigned to it. In the function body, this becomes available and can be modified (here, I simply add a key in it). The second function is a function expression, that gets assigned to a method of the prototype. When such an assignation is detected, the QML/JS plugin sets the prototype of the function expression to the prototype to which it is assigned. Here, SayHello.geet() and SayHello() share the same prototype. You can see that this.name points to the same declaration in both functions.

  

  This second screenshot shows how a function can be instantiated (instantiating a function? Javascript is very weird sometimes...). A variable named "greeter" is created, and its type is set to a StructureType. A StructureType represents a class instance in KDevelop, and can be used here to say that "greeter" is an instance of "SayHello".

  Now that "greeter" knows that it is an instance of "SayHello", and because I've added a bit of magick in how functions are handled when they are instantiated, the content of the prototype of "SayHello" becomes visible in greeter:

  

  This works fairly well and already allows complex libraries like impress.js to be somewhat usable, but there are still plenty of features missing, some of them being tightly bound to the dynamic nature of Javascript. For instance, "class inheritance" (or emulating it) is very complex and requires assigning a value to the prototype of a function. This could be supported, but ClassA.prototype = ClassB is not the recommended way of doing inheritance in Javascript. I've seen an example looking like this: ClassA.prototype = Object.createObject(ClassB.prototype);. The call to createObject hides the type of ClassB from the type deduction system of the plugin and greatly complexifies things.

  Even more advanced constructs, like assigning a function to the constructor property of an object, or playing with its __proto__ attribute, is nearly impossible to handle properly. I don't know if these kinds of constructs are widely used, but one single use at the base of a well-known Javascript framework can make the QML/JS plugin useless for the whole framework.

  The last point is that the QML/JS plugin makes no distinction between the prototype of a function and the objects intantiated from it:

  1 2 3 4 5 6 7 8

  function MyClass() { } var a = new MyClass(); var b = new MyClass(); a.foo = "bar"; // now b also has a "bar" attribute

  Solving this would be a bit complicated, and I think that such a behavior may even be desirable: imagine that QML/JS has missed the declaration of a member of "MyClass", because it was hidden in a function, used an unsupported construct, or anything else. The first time the user (or the library!) will try to use this missing member, it will be declared on the fly. This way, the user may even not notice that the real declaration of the member has been missed.

  QML directory imports

  Back to the nice world of staticly-typed and C++-ish QML :-). QML-related features are less challenging to implement, but QML is a very rich language that has many constructs that need to be handled. One example is QML imports, which exist in four flavors!

  • Standard module imports: import QtQuick 2.2 and import QtQuick 2.2 as QQ.
  • Automatic import of all the QML files in the current directory, so that a MyComponent.qml file can be instantiated by using the MyComponent component. No import statement needed.
  • Import of whole directories: import "../mydir"and import "../mydir" as MyDir. All the QML files in the imported directory are made available to the current QML files through their associated component names.
  • Import of Javascript files: import "../../js/module/myfile.js" as JSFile. Here, the "as" is mandatory. All the top-level declarations of the Javascript file (functions, variables, objects, etc) become available in a namespace named "JSFile" in my example.

  The first kind of import statement was supported for a long time. Support for the second type of imports has been added recently, and the last two types of import statements have been added today:

  

  Importing directories is also possible, but I did not have any interesting directory that deserved a screenshot. As always, every feature is unit-tested, and you can see the directory import feature at work in its unit test.

• Function call-tips and automatic declaration of object members

  Published on dim 06 juillet 2014 in Nepomuk, (Comments)

  I usually write a blog post once every week, but this time I have two features thay may already interest some people :-) . Now that the QML support in KDevelop is nearly complete and quite usable (several things are still missing though, for instance the support for directory imports), I wanted to work on Javascript-related things that I had to put aside for the past month or so.

  The first feature is function call-tips, that display the signature of the current function (or functions, if they are nested) as you type. This way, you can see the name and the type of the arguments, which can be very useful. This feature is available in Javascript and QML, and my example is, in fact, a QML code snippet.

  

  The second feature concerns only Javascript and is the automatic declaration of object members to which a value is assigned. Objects (or arrays) can be declared by two means in Javascript: using object literals (var o = {key: value, key2: value2};) and using an empty object literal, just to say that the variable is an object, and then each key is assigned its value separately:

  1 2 3 4

  var o = {}; o.key = value; o.key2 = value2;

  The two types of object declarations are widely used, the second one maybe even more than the first one. For instance, Javascript developers are encouraged to declare their functions in "namespaces", that are simply objects in the global scope :

  1 2 3 4

  var APP = {}; APP.say_hello = function() { alert("Hello"); }; APP.say_world = function() { alert("World !"); };

  The QML/JS plugin supports this by allowing the declaration of object members after the object itself has been created. This is fairly simple: when object.member is encountered in the source code, the plugin checks whether member already exists in object. If it is not the case, the member is declared on the fly and added to the object.

  Here are some technical details. The fact that the declaration of the member happens after the object has been completely created obliged me to cheat: KDevelop doesn't allow declarations to live outside their context (so I cannot declare the "say_hello" member of my example in the context that exists between the brackets of APP), but a context can import another one, even if the other one lives in a different file or anywhere else (this is what is used to implement import statements or C/C++ includes). The solution is therefore to declare say_hello in a new context that spans the "say_hello" identifier, and to import this context in APP. This way, "say_hello" becomes visible to APP, and code-completion works as expected:

  

  I'm now working on function prototypes and Javascript object-oriented programming. This is a bit more complicated than what I've thought, but I think I have found a possible solution. I will implement it tomorrow and I hope that it will make things like jQuery a bit more usable in KDevelop.

« Previous Page 2 / 6 Next »