Internship diaries #5

It’s the end of the internship experience and I feel lucky to have such a wonderful time at WSO2. I learned a lot on open-source technologies, middleware and their applications, ethics on JavaScript libraries, web services and most of all, problem solving, which had been contributing to the increase of my StackOverflow reputation as well.

As for extra-curricular activities, I learnt a lot about Table Tennis, Foosball and Carom and I even participated in the Tea-time championship as a part of Titans. The Secret Santa was one of the most exciting memories which include lunch outings and ice-cream outings.

As for the Data Mapper, the main work done during this phase was adding the format conversion functions.

Since the tree generation and the backend functionality was dealt with a JSON schema as the data, the data selected by the user had to be converted to JSON schema as well. The simplest was JSON to JSON schema, which was achieved by iterating over the keys and values in the JSON object to generate a matching JSON schema.

The CSV to schema conversion was achieved using an external library called mr-data-converter which converts CSV data to JSON which will be again converted to JSON schema by my previously written code.

XML and XSD conversion was the troublesome work since I was unable to find external opensource libraries with compatible licenses to use in the app. Most of the available libraries work with Node.js which again rules out the context of the app because the Data Mapper is supposed to be run in a static environment.

I achieved XML conversion using the HTML parsing functions and used its native methods to traverse through the XML. An interesting concept I came across during this conversion was the use of attributes in XML. And I had to come up with a key attributes to define the attributes in an XML content. For example,

<name id="345"> myName </name>

will be converted to:

name: {
        type: "string",
        attributes: {
                      id: {
                            type: "string"
                          }
                     }
      }

XSD conversion too had its own troubles with its namespaces and type definitions. XSD was a whole new topic for me and I learnt quite a lot about the tag names and their usages. During the conversion, I used a function where the first iteration would save the definitions and determine the root element, and a second iteration would traverse the root and build the schema.

Once the conversion was completed, I tested the functionality using SalesForce sample files which gave satisfactory results.

After taking the Data Mapper to a halt, I started looking into web services, especially RESTful web services. I was assigned with a task to develop a use case by incorporating a web service using JAX-RS and WSO2 products such as ESB, DSS and API manager. Currently I have worked around DSS to bind to a MySQL database, a web service to read information and I'm still working on connecting them using the ESB.

This internship experience has been such a wonderful time in my life. I learnt a lot on coding and working ethics, I improved my soft skills, especially on communication criteria, I got to meet some interesting people, share the knowledge as well as some good laughs. But all good things must come to an end. Only God knows what might be waiting for me in the coming new year, and I believe that the goodness will keep coming in. Thank you and Good Bye WSO2.

Internship diaries #4

At the start of the 4th four-week period, a review meeting was held where several new tasks were specified. I started working on them starting from the minor tasks.

The connector make over

One such task was converting the connectors from straight lines to lines with margin-like ends.
Earlier, the connector was simply an svg:line where I update the x1, y1, x2 and y2 properties directly using d3.selector.attr. In the new requirement, an svg:polyline was introduced as the solution where the points had to be updated instead of the previous x and y coordinates. Each polyline was of 3 segments and needed 4 pairs of coordinates. The earlier connectors already had 2 pairs of coordinates and I used the following to calculate the remaining 2 pairs.

In order to improve clicking on the connector, I added another polyline with the same points but higher width and invisible in the editor except for mouse hover. The events regarding the connectors were set to be captured by this underlying polyline and the increased area improved the usability of the connectors.

Clicked right

Another task was to add the load-file and clear containers functions to a right click menu on the container. For this I used predefined menus in <ul> tags in the main file, which were toggled on contextmenu event. There was a problem causing the menus to be displayed off the desired positions. After researching on this I found out the cause and the solution.

Lists (both unordered and ordered) have padding and margin set by default which was causing the undesired shifts. The default padding was meant for the bullet icons or numbering of list items. The simple solution was to set padding and margin to zero in the <ul> of the menu.

Hidden foreigner

Another problem faced during right click menu implementation was that the menus were not working for upper nodes of the container. After testing on multiple scenarios I found out that this occurs in Firefox but not Chrome and that the inspect-element pointed to a blank square rather than the nodes. The research on this showed that the <foreignObject> I used to hide the file-input was literally covering the upper nodes. Although I have set display:none to the input, I had not hid the <foreignObject> itself which was 100px wide and tall. To solve it, I set display:none to the parent element and as an extra precaution, set the height and width to 1px.

Build on the go

Among the major tasks I completed were dynamic addition of the nodes to the tree, which was also a use case of allowing the user to build a schema manually. I started on it by adding the root element. I used BootstrapDialog to create dynamic modal dialog boxes that gets the information such as the title and the type of the node. When the root element is added, a schema is initiated and stored in the Tree object.

The rules assumed as optimum are as follows:

1. The options for the type of the root element are limited to Object and Array
2. Only nodes of type Object or Array are allowed to add child nodes
3. New nodes to the root element are added only as children nodes
4. Sibling added to a node will be placed right after the triggered node
5. Child added to a node will be placed as the last child of the triggered node

When I was working on updating the schema during node addition, I noticed that JS does not allow adding new nodes to a specific location (to be added as the immediate sibling) since “JavaScript objects are unordered lists of properties”. Therefore a custom method was to be used to set out the right order.

I created a new temporary object, looped through the keys of the parent and kept adding them to the new object. When the triggered node is detected, I added the new sibling node and kept adding the rest of the nodes and finally replace the parent object with the new object.

Once the schema update was completed, I worked on displaying the new nodes. I wrote two (for sibling and child use cases) recursive functions to detect the node right-above the desired location. The functions were based on the HTML DOM hierarchy of the browser. Then I pushed down the nodes below the desired location and added the new node.

One of the problems I faced was that, the new nodes were added at the end of the HTML DOM hierarchy regardless of the position of the node in the tree, which gave incorrect results. Therefore a mechanism was used to re-order the elements in par with the schema, which was implemented using jQuery’s append and detach.

It’s the end of the 16th week of the internship and this post explains the progress of the JavaScript based data mapper so far.

After the ‘backbone’ of the app was set out, focus was given to adding more UI functionalities – where the user can customize the input, output and the operations.

The front-end

So far, the data mapper initializes with an input container, output container and a tool palette(integrated from the NEL tooling platform).

Each tool in the tool palette can be dragged and dropped on the canvas, where the corresponding operator is drawn. Each operator has its defined set of input/output labels and valid types and each operator has the option to be deleted.

Additionally, the direct connectors (Connecting Input container to output container) will be corresponding to a “Direct operator” without visible components.

The “Input” tool in the “Tree containers” is used to drag and drop extra input tree containers.

Each tree container has a total of 3 options which may appear or not based on the content in it. Empty containers have the option “Add new root” in place of the “Clear container” option in non-empty containers and the abstract functionalities of the options are as follows:

1. Load schema: Select the structure based on a file of one of the types of XML, JSON, CSV, XSD, JSON Schema.
2. Add new root: Build the structure manually by defining the root and adding nodes.
3. Clear container: Empties the container

The extra containers added by the user have the additional option to be deleted.

When reading from a file or building the structure in a tree container, the containers create “Node”s in a tree-structure. The parent nodes are of type Object or Array and others are leaf nodes and each leaf node has an “Anchor”. Input type anchors can be dragged and dropped on a node of opposite type which will create “Connector”s. A hidden svg:polyline of higher width is used to improve clicking on the connector.

Each node has 3 options as follows:

1. Add node: Add a new node to the structure.
2. Edit node: Edit the title and type of node
3. Delete node: Delete the node from the app.

When adding a new node, it is added as a sibling by default, but if the triggered node is a parent node, there is an option to add the new node as a child. When adding a child node, the child is added as the last child of the triggered node and when adding a sibling node, the sibling is added as the immediate next sibling of the triggered node.

The back-end at a glance

The tree structure of each tree container is maintained as a JSON Schema. When reading a file of a different type such as XML, JSON, CSV and XSD, the structure is converted to JSON Schema type and is manipulated throughout the app.

As a whole, the app creates and maintains a global JS object “Diagram” which includes the objects of the diagram as Backbone models or collections.

1 2 3 4 5 6 7 8 9

var Diagram = { Canvas: {}, //the editor = svg TreeContainers: {}, /* Backbone collection corresponding to Input/Output container the model contains the loaded schema as a JS object and a collection of nodes */ Connectors: {}, //all the 'lines' drawn in the application ToolList: {}, //The collection of tools PalleteView: {}, Operators: {} //a Backbone collection of all the operators in the canvas };

The result

The generated graph of mappings is calculated by means of the input/output adjacency lists of each operator. The result includes:

1. A variable list: All the nodes
2. Operators: A list of all operators including direct operators
3. Input adjacency list: Input nodes of the operators in the list
4. Output adjacency list: Output nodes of the operators in the list

Internship diaries #3

By the end of the 12th week, the data mapper tool front end was almost complete. And I had learned a huge list of functionalities of d3 as well as Backbone. It’s quite amazing how BackboneJS can provide an object oriented approach to Javascript. My application was completely converted into a bunch of models, collections and views interacting with each other.

The models of the app are as follows:

1. Operator – The container/operator box with a collection of nodes
2. Container(extending Operator) – Input/output containers
3. Node – The elements of the container. A node has a text, a text type, node type(object, array or leaf) and a type(input or output) which decides the placement of anchor.
4. Anchor – The arrow heads that are assigned to leaf nodes. Anchors of type output are draggable while the others are not.
5. Connector – The line that connects nodes through anchors. Each connector has a source node, source container, target node and target container. 

The collections of the app:

1. Operators – a collection of all the operators. The drawContainer function is responsible for drawing a container with the specified input count and output count on the canvas.
2. Connectors – a collection of all the connectors added to the app. The functions include findFromSource and findFromTarget which intakes a node and returns the connectors which have the specified node as the source or target respectively.
3. Nodelist – each container has a collection of nodes. The functions include getNodeFromDOMObject which returns the node  in the collection when the DOM object is specified.

Views initialize the app itself and the important functionalities. 

1. Canvas - The app is initiated by the CanvasView which initializes the required views and models of the app.
2. LoadFile – The file handling is managed by the LoadFileView which reads the specified file and draws the tree structure in the container.
3. OperatorPanel – This view is responsible for binding events to operator addition.

The mappings

At the event of calculating the mappings, the app outputs four arrays:

1. Variable list – an array of nodes. The id of the node in the array is used for the adjacency list.
2. Operators – a list of operators in the model, including direct operators.
3. Input adjacency list – each entry of the array corresponds to each operator and the inputs connected to the operator. Assumed that each node has only one input.
[input-nodes-of-op1, input-nodes-of-op2, ….] →  [2, 3] , [1, 2] …..
4. Output adjacency list – each entry of the array corresponds to each operator and the outputs connected to the operator. Assumed that each node can have multiple outputs, which is again an array.
[ [output-nodes-of-op1, output-nodes-of-op2, …... ]  → [ [9], [10, 11] ] , [ [7, 8], [10] ]….

Internship diaries #2

By the start of the second four-week period, the static web simulation of the Data Mapper tool was achieved by using JS Plumb but when the it required long codes to get the functionality work. Therefore, my mentor discussed with the other team members on an appropriate library to work on in a long term basis and came up to a conclusion on using d3. D3 is a simple yet powerful JavaScript library that provides many useful functionalities in handling document objects.

I was going through a tutorial on creating a bar chart using d3 and met with a a series of similar bugs. Looking into the matter by debugging, I found out that the cause of the bugs were version mismatches. I was using the latest version(v4) in my implementation while the tutorials I was going through were based on a previous version(v3). In order to recover from the bugs, I had to look into the details of version updates, and to my surprise there were quite a long list of structural changes in the version update.

The Data Mapper implementation using d3 started and in the mean time, I was researching on other similar JS libraries such as Jquery, Jquery-UI, AngularJS, NodeJS, BackboneJS and worked on sample projects through tutorials. I hadn’t even heard of some of them before. Additionally, I had to go through HTML SVG components as a majority of D3’s drawings were based on SVG.

The use cases of the JS based Data Mapper were:

1. File loading
2. XML parsing
3. Displaying tree structure
4. Implementing draggable containers
5. Implementing draggable nodes 
6. Detecting node drops.  

Implementation of file loading and XML parsing were reused from the earlier implementation using jsPlumb. Tree structure display was achieved by using a recursive method on which the node dragging functionality was implemented using D3’s drag functionality. The node drops were detected by using a method that calculates the areas of the leaf nodes of the opposite container.

Enabling dragging on containers was the trickiest objective which gave me an opportunity to learn about D3’s mouse events. During line dragging, the positions were calculated using mouse pointer position and the transform:translate attributes of the containers. I was practically living inside StackOverflow. After going through so much trouble, the container dragging functionality was finally solved. And it was time for a new task...

Internship diaries #1

This is the first of my monthly diaries of my internship experience at WSO2 Lanka (Pvt) Ltd.

An orientation session was carried out in the first week where we were introduced to the company hierarchy, organizational levels, rules and ethics, the products of the company and the resources available to use which can be used to improve our knowledge on the technical aspects. I personally find the culture of the company uniquely interesting where each employer is free from strict rules but everybody is genuinely passionate about what they are doing.

The products of WSO2 are open-source and they are based on the carbon kernel which serves as the core of each product. Components are integrated with the core to develop features. A product is basically a set of features assembled together to achieve a single or a set of specified functionalities.

In the first two days of the second week, project demonstrations were held from which we were asked to list our preferences. I selected the project named “Developing a data mapping tool plugin for VSCode” which was under team Developer Studio. My assigned mentor is Mr. Jasintha Dassanayaka. For the rest of the second week, I had a hands-on experience with the data mapping tool plugin for Eclipse.

The third week was spent on analyzing the source code of the data mapping plugin tool for Eclipse. The main features analyzed were the objects, their interactions, method calling sequences, file loading implementations and schema saving implementations.

The fourth week was spent on developing a static web app for data mapping tool using JavaScript. The implementations of file drag and drop, XML file parsing and the mapping functionality were mainly focused and completed successfully. To implement the mapping functionality, many libraries such as GoJS, jqTree, jsPlumb were researched but used jsPlumb due to the abundance of tutorials and community forums.

The problems faced during this period were overcome by the support of my mentor and on-line resources such as StackOverflow.