PLOMO2: A new generation of tools to navigate and profile programs

An INRIA Equipe Associee between RMOD and PLEIAD, a continuation of the PLOMO Equipe Associee named after cerro El Plomo, the highest peak (5424 meters) that is visible from Santiago. Foto by by Thiago "James".

Introduction

Performing effective software development and maintenance are best achieved with effective tool support. Provided by a variety of tools, each one presenting a specific kind of information supporting the task at hand. The goal of the first PLOMO was to develop new meta tools to improve and bring synergy in the existing infrastructure of Pharo (for software development) and the Moose software analysis platform (for maintenance). With Plomo2, we want to build on top of this work and invent a new generation tools to navigate and profile programs.

The hypotheses that Plomo2 will seek to verify are:

The overall objectives of Plomo2 are:

  1. Infrastructure for profiling programs and recording programmer activity.
  2. Visual software maps defined in a flexible and agile fashion
  3. Combining dynamic information with visualization to improve the development environment
  4. Empirical evaluation of this environment

All the efforts will be performed on Pharo and Moose, two platforms heavily used by the RMoD and Pleiad teams.

Work plan

Work Package One:

For both for program profiling and programmer activity recording, we need a way to deeply hook into the system. We plan for this to improve the reflective infrastructure of Pharo.

During the previous period we worked on OPAL, a new compiler toolchain for Pharo. In essence the work on the compiler framework does not move forward the state of the art in compiler technology. However it has already been proven to be a crucial building block. It provided the fundamentals for many research experiments and new features for Pharo. For example, Pleiad and RMoD collaborated on a novel type system for Pharo that was realized using the new compiler infrastructure. Other examples are the many new features of Pharo 3 that use parts of the new compiler chain, like AST-based navigation in the editor or breakpoints.

The Opal flexible compiler infrastructure will allow us to build a new generation of reflective infrastructure. Such infrastructure is keys to support tools (profilers) and new language design (AOP, proxies, isolation). We are now in a position to perform a new iteration on the reflective layer and metaobject protocol for Pharo. In 2005 we started to develop new reflective foundations and we learned a lot (partial reflection, metalevel recursion handling, object-centric on the fly propagation, proxies, isolation). We are now in a situation to step back and redesign the reflective layer based on all the cases we identified and developed.

The results of this work package will be used and thus validated by WP3. In addition, we plan to realize the next generation of the AOP framework Phantom using the reflection framework. This especially makes sure that the framework is general and not just limited for the use in tools.

Work Package Two:

Plomo made a lot of progress by realizing the Roassal visualisation framework. We plan to enhance the infrastructure to visually render software-related data. We will investigate two complementary directions:

  1. Interaction and scalability in Roassal 2D. Roassal is an Agile Visualization Engine that offers large range of options to make the user interact with the visualization. Visualization software require particular customizations to cope with large amount of data. Navigation should support interactive and semantic zooming, which is an essential feature for navigating large amount of data elements. Additional layers, including drawing dependency matrices, vectorial and interactive charts are required
  2. Visual representation of 3D environment gives a sensation of immersion that is often key to identify some patterns in a data structure. Current programming environments exploiting 3D are either fixed to a particular representation or unfit to be used as the base of a programming environment

Work Package Three:

The goal of the third work package is to build on top of both WP1 and WP2: Combining dynamic information with visualization to improve the development environment.

  1. Visualization to navigate and browse source code – System complexity is a polymetric view efficient at representing class hierarchies in a software source code. It has been shown that system complexity is effective for software maintenance. We will augment the range of application of it by using it to navigate and browser application source code.
  2. Visualizing dynamic information to navigate source code. The static view of code should be enhanced with dynamic information gathered from program execution. A good example here is visualization of profiling and tracing information.
  3. Context based on history – It has been shown that analyzing past user actions may precisely infer what will be the next user actions. Unfortunately, history is poorly supported in traditional programming environment. Making history a first class element in the programming environment is the topic of this work package.

This work package will therefore validate both WP1 and WP2. We will validate the ideas developed as part of WP3 by carrying out an empirical evaluation.

Achievements

Scientific Results

WP 1: Reflective Foundations

Progress on a new iteration on the reflective layer and metaobject protocol for Pharo: We have implemented the Slots Mechanism in Pharo, which resulted in a first-class reification of instance variables and their accessing through assignment and reading. This allows us to hook into reading and writing of instance variables and perform meta-level operations on them, e.g., adding breakpoints to the reading of a specific instance var, or a transparent notification mechanism for when a specific instance variable is written. The latter can be used, e.g., in WP3 to dynamically update a visualization of the object that contains the instance variable whenever the variable changes. Reflective operations in the form of meta-links can be added to any expression tree. All variables can now be annotated with meta data. Reflectivity has been improved to allow Meta- Links to be put on First Class Variables. All work has been integrated in Pharo5 development and is used already by other projects of RMOD and a research project of the University of Lugano/Switzerland. The stay of Marcus Denker in Chile in November 2015 had the goal to make the new reflective infrastructure to be reliable enough for other PLOMO sub projects to use (namely LRP and Profilers). A breakpoint system based on Metalinks has been added to Pharo5 and is being improved for the release of Pharo6. The object layout functionality of Pharo allowed us to work on supporting C-Types. The Opal Compiler was extended with primitive types and to emit new low-level bytecode instructions. Combined, this allows performance for use-cases like 2D and 3D frameworks to be speed up by 50-400% [13].

WP 2: Visualizations Engines

Two complementaries axes were investigated. Roassal 2D is a visualization engine to render data. Roassal 2D is employed to visualize any arbitrary set of data, including software-related data [5, 9] (e.g., source code, module dependencies), numerical set of numbers [6] (e.g., metric values). Roassal 2D has been improved from the interaction and scalability point of view. Roassal 2D is regularly used to visualize software systems large of thousands of classes. The second axis is the visualization in 3D. For that purpose, we have improved the FFI support of Pharo to use external libraries. With that, we created binding to SDL2 which allowed to move the window management code from the Virtual Machine to Pharo (OSWindow). The 2D infrastructure work (OSWindow) has been taken up by another Project (Inria in collaboration with Thales). It has been extended with multi-touch support and a gesture engine. Besides pre-defined low level gestures (e.g., pinch or multi-finger swipe), the engine is extensible with user-defined gestures. The Woden 3D graphics engine is a first demo that uses this foundation to bring the ideas of Roassal to 3D. The new next generation graphics frameworks like Metal and Vulcan where the driving force to revisit Woden. The new framework Woden 2 uses the lowcode subsystem (see WP1) for a new generation, high perfomance 3D framework for Pharo. This will be the foundation of Roassal3D soon. Woden has already seen use outside of the PLOMO project, for example it is used for experiments with virtual reality at a large french multi-national company.

WP 3: Tools for Analyzing Dynamic Information

Live Robot Programming (LRP) [10] as a case for the first two points of WP3: Visualization to navigate and browse source code, Visualizing dynamic information to navigate source code. Firstly, in LRP the (Domain-Specific) source code is shown in a (Domain-Specific) visualization made in Roassal2 and Spec. Clicking on the entities in the visual representation causes a navigation action in the source code of the program. Secondly, in the LRP visualization the dynamic information of the state of variables in scope is shown, as well as which piece of the code is currently running, and this is partially enabled by Slots. The visit of Marcus Denker to Chile in spring 2015 helped to make serious progress on the base implementation infrastructure: The LRP prototype now is based in the First Class Variables (Slots and Globals) that are available in Pharo. We worked on refining the user interface framework Spec [4,16], together with its integra- tion with the visualization of dynamic information using Roassal as well as the GT Inspector framework for the analysis of live objects. The goal is to allow a developer to build his own custom user interface for the visualization of dynamic information, easily combining classical UI widgets with more advanced visualizations (provided by Roassal and/or the powerful object inspection and modification abilities of the GT Inspector). One example of a specific UI built this way is the work of Fabry and Sinclair [14] allowing the analysis of the dynamic evolution of simulated object trajectories in different physics engines e.g., the Siconos engine by INRIA Bipop. This work on Spec is enhanced by the writing of extensive documentation on the frame- work, which was shown to be necessary for developers to effectively be able to construct such user interface

Work outside of work packages

Close proximity among the researchers also allowed work outside of the work packages to be started. One example is a preliminary effort in speeding up collections handling [12], another the work on identifying classes in legacy JavaScript code [15].

Publications

  1. Johan Fabry, Andy Kellens, Simon Denier, Stephane Ducasse. “AspectMaps: Extending Moose to Visualize AOP Software”. Science of Computer Programming, volume 79, number 1, pp.6-22, January 2014, Elsevier https://hal.inria.fr/hal-01086997
  2. Johan Fabry, Romain Robbes, Marcus Denker. “DIE: A Domain Specific Aspect Language for IDE Events”. Journal of Universal Computer Science, volume 20, number 2, pp.135- 168, February 2014 https://hal.inria.fr/hal-00936376
  3. Esteban Allende, Oscar Callau, Johan Fabry, E ́ric Tanter, Marcus Denker. “Gradual Typing for Smalltalk”. Science of Computer Programming, volume 96, number 1, pp.52- 69, December 2014, Elsevier https://hal.inria.fr/hal-00862815
  4. Benjamin Van Ryseghem, Stephane Ducasse, Johan Fabry. “Seamless Composition and Reuse of Customizable User Interfaces with Spec”. Science of Computer Programming, volume 96, number 1, pp.34-51, December 2014, Elsevier https://hal.inria.fr/hal-00915350
  5. Alexandre Bergel, Sergio Maass, Stephane Ducasse, Tudor Girba. “A Domain-Specific Language For Visualizing Software Dependencies as a Graph”. Proceedings of 2nd IEEE Working Conference on Software Visualization (VISSOFT NIER) https://hal.inria.fr/hal-01369700
  6. Popular Article LinuxMag: Roassal and the Charter charting framework. http://www.gnulinuxmag.com
  7. Miguel Campusano and Johan Fabry. “From Robots to Humans: Visualizations for Robot Sensor Data.” In the third IEEE Working Conference on Software Visualization (VIS- SOFT 2015), Sep 27-28, 2015 , IEEE press. http://dx.doi.org/10.1109/VISSOFT.2015.7332424
  8. Damien Cassou, Stephane Ducasse, Luc Fabresse, Johan Fabry, Sven Van Caekenberghe: Enterprise Pharo. This is the third volume of a series of books covering Pharo and its community. http://files.pharo.org/books/enterprise-pharo/
  9. Book: Alexandre Bergel, Agile Visualization: http://agilevisualization.com
  10. Miguel Campusano and Johan Fabry: “Live Robot Programming: The Language, its Implementation, and Robot API Independence”. Science of Computer Programming, Elsevier. Received 8 September 2015, Revised 28 March 2016, Accepted 9 June 2016, Available online 17 June 2016. http://dx.doi.org/10.1016/j.scico.2016.06.002
  11. Leonardo Humberto Silva, Marco Tulio Valente, Nicolas Anquetil, Daniel Carlos Hovadick Flix, Alexandre Bergel, Anne Etien. “JSClassFinder: A Tool to Detect Class-like Structures in JavaScript”. Proceedings of Brazilian Conference on Software: Theory and Practice (CBSOFT’15) https://hal.inria.fr/hal-01369705
  12. Juan Pablo Sandoval Alcocer, Marcus Denker, Alexandre Bergel, Yasett Acurana. “Dynamically Composing Collection Operations through Collection Promises’. In: Proceed- ings of IWST16. https://hal.inria.fr/hal-01358347
  13. Ronie Salgado, Stephane Ducasse “Lowcode: Extending Pharo with C Types to Improve Performance”. In: Proceedings of IWST16. https://hal.inria.fr/hal-01353884
  14. Johan Fabry and Stephen Sinclair: ”Interactive Visualizations for Testing Physics Engines in Robotics”. Proceedings of 4th IEEE working Conference on Software Visualization (Tool paper) 3-4 Oct 2016. IEEE Computer Society Press
  15. Leonardo Humberto Silva, Marco Tulio Valente, Alexandre Bergel, Nicolas Anquetil, and Anne Etien “Identifying Classes in Legacy JavaScript Source Code’ In submission to JSS
  16. Book in preparation about Spec. https://ci.inria.fr/pharo-contribution/view/Books/job/BuildingUIWithSpec/

Software artifacts

Event organization

Visits

Santiago To Lille:

Lille to Santiago: