PLOMO: Customizable Tools and Infrastructure for Software Development and Maintenance

An INRIA Equipe Associee between RMOD and PLEIAD, named after cerro El Plomo, the highest peak (5424 meters) that is visible from Santiago. Foto by by Thiago "James".

Software maintenance is the process of maintaining a software system by removing bugs, fixing performance issues and adapting it to keep it useful and competitive in an ever-changing environment [Chik90a]. Performing effective software maintenance and development is best achieved with effective tool support, provided by a variety of tools, each one presenting a specific kinds of information supporting the task at hand [Deme02a].

The goal of Plomo is to develop new meta tools to improve and bring synergy in the existing infrastructure of Pharo (for software development) and the Moose software analysis platform (for maintenance). Plomo will

1. enhance the Opal open compiler infrastructure to support plugin definition,
2. offer an infrastructure for change and event tracking as well as model to compose and manipulate them,
3. work on a layered library of algorithms for the Mondrian visualization engine of Moose,
4. work on new ways of profiling applications.

All the efforts will be performed on Pharo and Moose, two platforms heavily used by the RMoD and Pleiad teams.

Opal is a new, extensible and modular compiler framework. The front-end is based on the notion of dynamically composable parsers [Reng10c] and a high-level code representation based on an AST (abstract syntax tree) with code transformation support [Robe96a]. The backend is based on an intermediate representation of byte code. Such a high-level representation of byte code offers a good possibility to express advanced, on the fly, modification of methods at the byte code level. A flexible and extensible byte code level infrastructure is needed to introduce code for hooks and low level probes, as needed by advanced aspect oriented languages (AOP) or dynamic analysis tools like runtime profilers. The compiler framework was developed as a joint project between RMod and the SCG in Bern. Opal acts as the infrastructure behind several other work packages and needs to be enhanced to enable full support for those.

The goal of this task is to be able to replace the existing compiler infrastructure in place in Pharo by Opal, allowing the other tasks of this proposal to use Opal as a basis. Replacing the old Pharo compiler validates the compiler as practically usable.

Many development environment tools provide additional support to the fundamental edit-compile-run cycle of development and maintenance. These range from basic, state-of-the-practice tools such syntax highlighting, code autocompletion, or notification of changes, to experimental tools such as task context management [Kers06a] and code clone detection and clone management. These tools need to be integrated in the IDE in a fluid manner.

These additional tools hook into the core of the IDE, adding their notification system for the actions taking place inside of the IDE. The events that they track can differ, ranging from events produced by the IDE (changes to the code, navigation in the code, tool usage, copy/paste, etc.) to events produced by the program that is being developed at runtime, as generated by debugging or profiling tools. Currently, these additional tools contain an ad-hoc event handler, i.e., each tool implements its own event handlers and event generators. As a result, there is low reuse in that part of the construction of such additional support tools, and furthermore the different event handlers in the system may interfere in each others' operation. The various event generators have heterogeneous interfaces as well, so that a tool needing to handle several kinds of events is exposed to additional complexity.

The goal of this task is to develop a general event notification system for Pharo and building a set of suitable abstractions on top of this system. We plan to extend and refine the current Announcement system, and reify events at the level of abstraction that allows the additional support tools to express sequences of events of interest in a concise way. Conceptually this is a case of event-based Aspect Oriented Programming [Doue02b]. The IDE generates a stream of events which are join points, the abstraction level is obtained by a (higher-level) join point model, and the tools express their interest in a certain kind of stream of events by declaring a pointcut. Developing a Domain-Specific Aspect Language for IDE events will allow tool builders to declare event handlers in a concise and homogeneous way, allowing tools to express interest in different stream of events seamlessly.

Software visualization is an essential component of many reverse engineering and software analysis approaches. The first step for this research axis is to produce the necessary infrastructure for conducting analysis and experiments using software visualization. We envision a framework that is open and malleable for a broad range of visual representations. This work package is built on top of the Mondrian visualization framework. Mondrian has been successfully employed in numerous academic and industrial case studies. It has demonstrated its ability to produce useful and practical high level source code representations, helpful to identify bugs and anomalies in software programs [Lava09a,Berg10b,Berg10a]. Although expressive, Mondrian suffers from its built-in graph model to represent data. It is often a constraint since its restrict creativity and scalability (especially when low and high level of details are simultaneously displayed). As the result of this first work package,we plan to produce Mondrian2 (working title for the new version of Mondrian), an open-source framework for agile, flexible, scalable and open visualizations of both static as dynamic information. For the latter, Mondrian will be able to take advantage of the new event mechanism implemented in the second workpackage to allow for dynamic visualizations of behavioral/run-time information.

Application profiling can be realized for a variety of reasons (locating performance bottlenecks, assessing test coverage, to support feature location, etc.). Each use case requires different strategies to gather the necessary information, and has different performance requirements.

In this work package we will focus on providing the necessary infrastructure for profiling applications in such a variety of situations. There are several scientific challenges such as collecting, storing events in memory efficient manner.

Such a framework ought to be flexible and designed for rapid prototyping of profilers. The key asset of this framework is to be open for dynamic extension and refinement: profilers will be adjustable while acting on an executing software program. This ability, which we call agility, is essential to enable an effective profiling in a dynamic programming language and environment such as Pharo, the programming language we have chosen for conducting this work. To effectively exploit produced profilers, the framework has to be fully integrated with the IDE of Pharo. As an outcome, this work package will produce an agile and effective framework for designing and realizing program execution profilers.

Here we present in a condensed way the achievements of each workpackage first by work package and then by years.

The compiler framework was used to build a type system for Pharo: Gradualtalk [Alle13a]. A gradually-typed Smalltalk allows one to enable incremental typing of existing programs. The main design goal of the type system is to support the features of the Smalltalk language, like metaclasses and blocks, live programming, and to accommodate the programming idioms used in practice. We studied a number of existing projects in order to determine the features to include in the type system. As a result, Gradualtalk is a practical approach to gradual types in Smalltalk, with a novel blend of type system features that accommodate most programming idioms.

In the context of the Gradualtalk type system, we have further developed, debugged and released the compiler framework. Opal has been integrated as the new compiler for Pharo [Bera13a]. It is stable, robust and is the new default compiler for day to day development in Pharo3. Opal provides the basis for many new features in Pharo3 and provides a new foundation for building new layers such as an advanced reflective model.

Opal solves three main problems of the old compiler infrastructure:

1. The architecture is not reusable
2. compiler can not be parametrized
3. the mapping between source code and bytecode is overly complex.

In essence the work on the compiler framework does not move forward the state of the art in compiler technology however it has already been proven to be a crucial building block. It provides the fundamentals for many research experiments and new features for Pharo. The work on the type system is one example, others are the many new features of Pharo3 that use parts of the new compiler chain, like AST based navigation in the editor or breakpoints.

The Opal flexible compiler infrastructure will allow us to build a new generation of reflective systems. Such reflective systems are keys to support tools (profilers) and new language design (AOP, proxies, isolation). We are now in a position to perform a new iteration on the reflective layer and metaobject protocol for Pharo. In 2005 we started to develop new reflective foundations and we learned a lot (partial reflection, metalevel recursion handling, object-centric on the fly propagation, proxies). We are now in a situation to step back and redesign the reflective layer based on all the cases we identified and developed.

We developed EPICEA a new model of changes and an implementation representing all the changes made during development [Dias13a] - By changes we mean: method, class, package definition, modification, removal but also new coding session, refactorings … It is the basis for a large number of analyses (cherry picking, code review support, replay of sequences, code recovering, browsing in the past) and tools that we will build around change management. In particular EPICEA will be extended to support branch merging and propose new analyses to help developer taking merging decisions.

Complementary to that, we developed DIE, a Domain-Specific Aspect Language that provides a set of domain-specific abstractions for building plug-ins to a development environment. It allows tool builders to declare event handlers in a concise and homogeneous way, allowing IDE extensions and tools to express interest in different streams of events seamlessly, with a consistent syntax regardless of the source of the event.

Over the last 3 years we have developed Roassal, an agile visualization engine. We have dropped the name Mondrian 2 in favor of Roassal. Roassal is made to visualize and interact with arbitrary data, defined in terms of objects and their relationships. Roassal is commonly employed to produce interactive visualizations. The range of applications using Roassal is diverse. For example, the Moose community uses Roassal to visualize software.

Roassal is a visualization engine working on 4 different platforms (Pharo, VisualWorks, Amber, VASmalltalk). Roassal has been adopted by several companies (such as LAM http://www.lamrc.com) located in Europe, Australia and North America. Maintenance of Roassal and porting efforts are supported by Object Profile. Roassal is a key element of the Moose open-source analysis platform developed by RMOD and the synectique company (http://www.synectique.eu) spinoff of RMOD.

Roassal is an engine used by several research group worldwide (RMOD, IRC (FR), SCG (CH), Reveal (CH),…) and the foundations of several research publications (profiler, memory manager, program visualizations,…).

The book Deep Into Pharo contains two chapters about Roassal.

Profiling techniques have been co-developed by RMoD and Pleiad. Understanding the root of a performance drop or improvement requires analyzing different program executions at a fine grain level. Such an analysis involves dedicated profiling and representation techniques. JProfiler and YourKit, two recognized code profilers fail, on both providing adequate metrics and visual representations, conveying a false sense of the performance variation root.

We propose performance evolution blueprint, a visual support to precisely compare multiple software executions. Our blueprint is offered by Rizel, a code profiler to efficiently explore performance of a set of benchmarks against multiple software revisions.

We worked on Spec: a UI Builder that proposes a new way to compose applications out of widgets [Ryse13a]. A missing aspect of existing UI Builders is the ability to reuse and compose widget logic. This leads to a significant amount of duplication in UI code. Instead, Spec focuses on widget reuse with widget properties that are defined declaratively and and attached to specific classes that are built specifically to be reusable.

We participated to the new release of Moose http://www.moosetechnology.org. The key highlights are:

• Based on Pharo 2.0.
• All built-in visualizations use Roassal.
• Roassal received a significant performance boost and new smart graph layouts like TreeMapLayout or ForceBasedLayout.
• Roassal uses Athens for nice looking vectorial graphics.
• New charting engine based on Roassal: Graph-ET.
• Glamour changed to enable dynamic scripts, while still remaining backward compatible.
• Glamour received a RubricTextPresentation for using the new Rubric text morph.
• Glamour, Roassal, EyeSee, Graph-ET editors are more robust when dealing with errors in scripts.- New GTDebugger with dedicated workflows for Announcements, PetitParser and Glamour itself.
• Extended GTInspector for several object types.
• Usable version of GTMetaceller for handling Metacello configurations.
• FAMIX was strengthen to handle functions better in various programming languages.
• New free type fonts and simpler whitespace-loving theme.
• Lower memory footprint for large models.
• Faster MSE import.

We participated to the new release of Pharo 1.4 and 2.0 and are actively developing Pharo3 (http://pharo-project.org). The speed of Pharo development is increasing with each version. For Pharo2, a list of changes can be found on the website. PLOMO members contributed many smaller and larger improvements found while doing the research work described in this report. When visiting, PLOMO members take care to organize Pharo Sprints, open meetings focussed on fixing bugs and integrating features.

A small example for how PLOMO directly contributed can be seen with the type system work. While type-checking the code of the Pharo system, a number of problems where detected. Examples where dead code, wrong inheritance relationships between classes and others. All these problems have been fixed in Pharo2.

For Pharo3, PLOMO contributed in a major way: the Opal Compiler replaced the old default compiler infrastructure.

Now we list the results per year and with different angles.

A number of results have been obtained during the first year of Plomo:

1. Roassal is an agile visualization engine. Roassal is used by the RMoD research group and is is primarily developed in Pharo.
2. Roassal is also used by the Moose software analysis platform.
3. Enhanced Opal to support type annotations. Opal is developed and maintained by RMoD.
4. ObjectProfile is a new company based in Chile. Its business plan is essentially focused on Pharo and Roassal.
5. Released Pharo 1.3 (http://www.pharo.org)

In the second year of execution of Plomo, work has focused on:

1. Rizel: a performance evolution monitor.
2. A book chapter on Roassal in the book Pharo By Example 2
3. Roassal also won the third place award in the ESUG 2012 innovation technology awards.
4. Development of Athens the graphic rendering engine developed by RMoD for Pharo. Athens is used by Roassal.
5. Starting of the founding process of Synectique, a company based in Lille that offers solutions based on the Moose platform. ObjectProfile offers to Synectique a dedicated support of Roassal.
6. Integration of profiling techniques into Jenkins, the continuous integration server used for Pharo. We expect to have a massive amount of profiling information.
7. Opal debugging and development continued. The bytecode backend is ready for integration in Pharo 2.0.
8. Gradualtalk: a gradually typed Smalltalk, built on Opal, has been implemented. It allows code in Pharo to be gradually and optionally typed.
9. The Announcements framework to enable change and event tracking.
10. Spec: a Framework for the Specification and Reuse of UIs and their Models. It uses the Announcements framework to enable fine-grained UI refreshes. Roassal makes use of Spec for its component
11. Work on the DIE domain-specific language and the definition of IDE plugins using it, as well as work on change prediction models are still ongoing.
12. Paper: “How (and Why) Developers Use the Dynamic Features of Programming Languages: The Case of Smalltalk”,
O. Calláu, R. Robbes, É. Tanter, D. Röthlisberger — EMSE, in press
Empirical Software Engineering (Springer)
13. Paper: “Extensions during Software Evolution: Do Objects Meet Their Promise? ”,
R. Robbes, D. Röthlisberger, É. Tanter — ECOOP 2012, 26th European Conference on Object-Oriented Programming
14. Released Pharo 1.4 (http://www.pharo.org)

1. Juan Pablo Sandoval Alcocer, Alexandre Bergel, Stéphane Ducasse, Marcus Denker – Performance Evolution Blueprint: Understanding the Impact of Software Evolution on Performance. Proceedings of 1st IEEE Working Conference on Software Visualization VISSOFT, 2013.
2. The book Deep into Pharo, after 4 years of hard work
3. Work on the DIE domain-specific language and the definition of IDE plugins using it was submitted to a journal and is in a second round of revisions.
4. Organization of a coding sprint at Santiago in January 2013 (12 participants)
5. Participated to Moose 4.8 release (http://www.moosetechnology.org).
6. Released Pharo 2.0 (http://www.pharo.org)
7. Integrated the Opal Compiler in the Pharo3 development branch.

1. Juraj Kubelka, PhD student Universidad de Chile.
2. Juan Pablo Sandoval, PhD student Universidad de Chile.
3. Martin Dias, PhD student, RMoD, Inria.

1. Synectique is a company delivering dedicated software analyses. Synectique uses Roassal to visually report customer source code analysis. The founding process started in 2012, and the company was created in June 2013.
2. ObjectProfile was founded in 2011 in Chile. Its business plan is essentially focused on Pharo and Roassal. Object Profile offers support of its products to RMoD and Synectique. A number of features of Roassal have been designed to meet Synectique's requirements (e.g., the navigation and scrolling options).

Plomo enables the participants RMoD (INRIA Lille Nord Europe) and Pleiad (the University of Chile) to collaborate by funding their trips.

• Marcus Denker from November 7th until November 28th, 2011
• Marcus Denker from January 18 until February 2nd, 2012 (2011 budget)

• Vanessa Peña and Alexandre Bergel, Aug 15 until Aug 20, 2011. From Aug 20 until Aug 28 they attended ESUG, a conference co-organized by RMoD. The topic of this research visit is test coverage and software visualization. Result of the research activity on software visualization and code profiling have been presented at ESUG.
• Romain Robbes from July 18 until July 24, 2011. Partially funded by Pleiad.
• Esteban Allende from July 19 until October 2, 2011. Esteban's stay was partially funded by the French Embassy in Chile.

• Benjamin Benjamin Van Ryseghem from May 29th until June 15th, 2012. Partially funded by Pleiad.
• Damien Pollet from November 1 until November 30, 2012. Partially funded by Pleiad.
• Marcus Denker from November 5 until November 22, 2012

• Johan Fabry from March 19th until March 23rd, 2012.
• Johan Fabry from August 17th until Sept 2nd, 2012. Fully funded by Pleiad
• Juan Pablo Sandoval from 9 November until 2 December 2012. The topic of the research visit is monitoring of performance evolution.
• Alexandre Bergel, Vanessa Peña, Juan Pablo Sandoval, Pablo Estefo from August 24 until September 2. All participated in ESUG, a conference co-organized by RMoD, where in parallel work has been performed together with RMoD on profiling applications. These trips were fully funded by Pleiad.

• Stéphane Ducasse from November 4 until November 15, 2013. Partially funded by Pleiad.

• Johan Fabry on 15th of July, 18th and 19th of September 2013
• Alexandre Bergel from December 12 until December 29, 2013 (to confirm)
• Alejandro Infante from September 13 until September 21, 2013
• Ronie Salgado in january 2014. (to confirm)

• Stephane Ducasse, Manuel Oriol, Alexandre Bergel, Challenges to support automated random testing for dynamically typed languages, In Proceedings of the 3rd International Workshop on Smalltalk Technologies (IWST'11), Collocated with ESUG, June 2011. ACM Digital Library http://dx.doi.org/10.1145/2166929.2166938
• Johan Fabry, Andy Kellens, and Stéphane Ducasse. AspectMaps: A Scalable Visualization of Join Point Shadows. In Proceedings of 19th IEEE International Conference on Program Comprehension (ICPC2011), pages 121–130. IEEE Computer Society Press, Jul 2011 http://dx.doi.org/10.1109/ICPC.2011.11.

• Johan Fabry, Andy Kellens, Simon Denier and Stéphane Ducasse. AspectMaps: Extending Moose to Visualize AOP Software, In Elsevier Science of Computer Programming, Special issue on Experimental Software Toolkits, 2012. http://dx.doi.org/10.1016/j.scico.2012.02.007
• Benjamin Van Ryseghem, Stéphane Ducasse, Johan Fabry, Spec: a Framework for the Specification and Reuse of UIs and their Models In Proceedings of the 4th International Workshop on Smalltalk Technologies (IWST'12), Collocated with ESUG, August 2012. ACM Digital Library http://dx.doi.org/10.1145/2448963.2448965
• Juan Pablo Sandoval, Tracking Down Software Changes Responsible for Performance LossIn Proceedings of the 4th International Workshop on Smalltalk Technologies (IWST'12), Collocated with ESUG, August 2012. ACM Digital Library http://dx.doi.org/10.1145/2448963.2448966

• Esteban Allende, Oscar Callau, Johan Fabry, Eric Tanter, and Marcus Denker, “Gradual Typing for Smalltalk” In Science of Computer Programming, accepted and available online. http://dx.doi.org/10.1016/j.scico.2013.06.006
• Paper: Juan Pablo Sandoval Alcocer, Alexandre Bergel, Stéphane Ducasse, Marcus Denker – Performance Evolution Blueprint: Understanding the Impact of Software Evolution on Performance. Accepted at 1st IEEE Working Conference on Software Visualization VISSOFT, 2013. online
• Deep into Pharo, 421 pages, Published by Square Bracket Association
• Dehouck Mathieu, Usman Bhatti, Alexandre Bergel and Stéphane Ducasse, Pragmatic Visualizations for Roassal: a Florilegium, in International Workshop on Smalltalk Technologies, 2013. online
• Martin Dias, Damien Cassou and Stéphane Ducasse, Representing Code History with Development Environment Events, in International Workshop on Smalltalk Technologies, 2013. online
• Clément Béra and Marcus Denker, Towards a flexible Pharo Compiler, in International Workshop on Smalltalk Technologies, 2013. online

• Romain Robbes, Johan Fabry, Marcus Denker, DIE: A Domain Specific Aspect Language for IDE Events, accepted with major revision at Journal of Universal Computer Science.
• Benjamin Van Ryseghem, Stéphane Ducasse, Johan Fabry, Alain Plantec Seamless reuse of customizable user interfaces with Spec, accepted with minor revisions in Elsevier Science of Computer Programs.
• Romain Robbes, Damien Pollet, Michele Lanza, Improving change prediction by replaying fine-grained development histories, In preparation.

[Chik90a] Elliot Chikofsky and James Cross II. Reverse Engineering and Design Recovery: A Taxonomy. In IEEE Software 7(1) p. 13-17, January 1990.

[Deme02a] Serge Demeyer, Stéphane Ducasse, and Oscar Nierstrasz. Object-Oriented Reengineering Patterns, Morgan Kaufmann, 2002.

[Reng10c] Lukas Renggli, Stéphane Ducasse, Tudor Girba, and Oscar Nierstrasz. Practical Dynamic Grammars for Dynamic Languages. In 4th Workshop on Dynamic Languages and Applications (DYLA 2010), Malaga, Spain, June 2010.

[Robe96a] Don Roberts, John Brant, Ralph E. Johnson, and Bill Opdyke. An Automated Refactoring Tool. In Proceedings of ICAST '96, Chicago, IL, April 1996.

[Lava09a] Jannik Laval, Alexandre Bergel, and Stéphane Ducasse. Matrice de dépendances enrichie. In Proceedings of Languages et Modéles á Objets (LMO 2009), Nancy, France, 2009.

[Berg10b] Julio Ariel Hurtado Alegria, Alejandro Lagos, Alexandre Bergel, and María Cecilia Bastarrica. Software Process Model Blueprints. In Proceedings of the International Conferences on Software Processes (ICSP'10), LNCS Springer Verlag, July 2010.

[Berg10a] Sébastien Mosser, Alexandre Bergel, and Mireille Blay-Fornarino. Visualizing and Assessing a Compositional Approach of Business Process Design. In Proceedings of 9th International Conference on Software Composition (SC'10), LNCS Springer Verlag, July 2010.