Live Robot Programming

Live Robot Programming (LRP) is a live programming language designed for the creation of the behavior layer of robots. It is fundamentally a nested state machine language built with robotics applications in mind, but it is not bound to a specific robot middleware, API or OS.

LRP is open source, download instructions are below, example code can be found online and there is a public bug tracker.

We all like videos of robots, right? Here are some of LRP on a few different robots for your viewing pleasure.

• How does a PR2 move through a door with LRP? (This uses ROS to control the robot.)
• Flying a Parrot AR.Drone 2, using AR tags. (This uses the Parrot API to control the drone.)
• Making a NAO robot follow a ball. (This uses the NAO API to control the robot.)
• PR2 pick and place interaction with speech control. (This uses ROS to control the robot.)

NEWS:

• Parrot AR.Drone 2 support is finished!
• A first version of NAO support is out!
• Journal paper defining LRP, the functioning of the interpreter and bridges to robot API accepted for Science of Computer Programming, a preprint is available.
• NIER Paper on visualizing robotic sensors accepted at VISSOFT 2015, a preprint is available.
• Johan presented LRP at the ESUG 2015 conference. A video of the talk is available as part of the conference playlist on Youtube.
• LRP was part of Johan's talk at the robotics day conference, claimed to be the largest robotics event in Latin America.

Have a look at one minute of LRP coding: making a state machine for the tick-tock of an oldfashioned analog watch:

As LRP is not bound to a specific robot infrastructure, bridges to a given infrastructure are add-ons to the language. The language can therefore also be used 'simply' as a nested state machine live programming language! Currently LRP provides a bridge to ROS and to the Lego Mindstorms EV3 via JetStorm. A bridge to the Parrot AR.Drone 2 is under development and will be released Real Soon Now .

LRP is implemented in Pharo Smalltalk, so basic knowledge of Smalltalk is required to be able to write useful code. But don't panic, Pharo itself comes with a set of tutorials that are sufficient for being able to use LRP.

LRP programs describe nested state machines: a state of a machine may contain (several) complete state machine(s), whose states may again contain a machine, and so on. Next to states, machines may also define transitions and events. Events are named actions (= Smalltalk blocks) that specify the conditions for transitions. LRP has four kinds of transitions:

• normal transitions: the usual ones
• epsilon transitions: who trigger automatically when the state is active
• timeout transitions: who trigger after their timeout (in milliseconds) given as a literal or as a variable reference
• wildcard transitions: who have no specific source state, they consider all of the states in their machine as the source state.

The following is an example state machine (called esc) that models the escapement of a mechanical clock (the part responsible for the tick-tock). It consists of two states (tick and tock) and the timeout transitions between them. To the right is its visual representation in the LRP editor.

(machine esc
	(state tick)
	(state tock)
	(ontime 500 tick -> tock)
	(ontime 500 tock -> tick)
	)

To start interpreting this machine, a spawn statement needs to be specified. Spawns can be declared either at the top level, for the top level machine or as the onentry action of a state (see below for onentry), for nested machines. The video at the top of this page shows the process of writing and running this code.

(spawn esc tick)

States may also contain actions: an on entry action (onentry), an on exit action (onexit) and a running action (running). Actions are Smalltalk blocks. When a state becomes active, its on entry action is executed once, completely. When the state stops being active, its on exit action is executed once, completely. While the state is active, its running action is executed as the main part of the interpretation loop. In other words, the block is executed repeatedly, and between each execution of the block the events are checked. ¹⁾

A machine can also define variables. Global variables may also be defined, outside of the root machine. Variables must be given a value when they are defined, the value is the result of evaluating a Smalltalk block. This block has in scope all variables that are lexically in scope. All actions (i.e. onentry, onexit, running blocks) may read, send messages to, and set all variables in their lexical scope.

For example, to add a seconds counter to the escapement, the code is as below. We show a screenshot of the complete editor here. It shows the code pane (left), the tree of machines and current variables (center), and the visualization of the machine (right).

Another example is a bit more complicated: a resettable timer that shows minutes as a variable, and seconds (rounded to 10) as a state. To the right of the code we show the relevant part of the editor. The go transition is an epsilon transition, the reset transition is a wildcard transition because of the * before the arrow replacing the source state.

;; a resettable timer with 10 sec intervals
(var minute := [0])
(machine timer
  (state zero)
  (state ten)
  (state twenty)
  (state thirty)
  (state fourty)
  (state fifty
    (onexit [minute := minute + 1]))
  (ontime 10000 zero -> ten toten)
  (ontime 10000 ten -> twenty totwenty)
  (ontime 10000 twenty -> thirty tothirty)
  (ontime 10000 thirty -> fourty tofourty)
  (ontime 10000 fourty -> fifty tofifty)
  (ontime 10000 fifty -> zero tozero)

  (var doreset := [0])
  (state init
    (onentry [minute := 0. doreset := 0]))
  (on resetting *-> init reset)
  (eps init -> zero go)
  (event resetting [doreset = 1])
)
(spawn timer zero)

Below is a video that shows this machine in action, and demonstrates that when the doreset variable is set to 1

• the wildcard transition occurs to init,
• whose onentry action resets minute to 0 and doreset to 0,
• which is then followed by the epsilon transition to zero.

As a variation on the clock theme, below is the example for a stopwatch. The values of the variables influence whether the stopwatch is running or stopped, and if it should be reset. Also, the stopwatch state machine is an example that uses all transition types available in LRP.

;;; Stopwatch
(machine stopwatch
  (var start := [false]) (var reset := [false]) (var seconds := [0])
  (state waiting )
  (on starting waiting -> tick)
  (event starting [start])
  (state tick)
  (state tock (onexit [seconds := seconds + 1]))
  (ontime 500 tick -> tock )
  (ontime 500 tock -> tick )

  (on reset *-> resetting)
  (event reset [reset])
  (state resetting (onentry [reset := false. seconds := 0] ))
  (eps resetting -> tick tet)

  (on stop *-> waiting tes)
  (event stop [start not])
)
(spawn stopwatch waiting)

LRP is available on SmalltalkHub, under the MIT license. To be able to use it, you first need to download Pharo 5 and basic knowledge of Pharo is required. Note that Pharo itself comes with the tutorials required to get you up to speed.

You can download a working image of the latest development version from our continuous integration server. This image is ready to work, containing LRP and any other project that LRP needs. In this image you will have a new menu entry in the World menu that opens the LRP UI. (Click on the background to get the World menu.)

We have example code online in our GitHub repository. To work with ROS or the Lego Mindstorms EV3 via JetStorm, download the respective bridge packages, as detailed below.

Installing LRP From Source

Alternatively, you can install LRP from source, do-it of the following in a playground:

Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  configuration;
  loadDevelopment

ROS integration is provided thanks to PhaROS. You can download a working image with LRP and PhaROS that is relatively up-to-date. This image is ready to work, you will have a new menu entry in the World menu that opens the LRP UI. (Click on the background to get the World menu.)

When the interpreter is opened, a separate UI allows for making subscriptions to topics and declaring topics on which to publish. For example code that uses ROS we refer to our GitHub repository.

From Source

We will assume you are using a Ubuntu linux, as this is the standard OS for ROS. To install from source, first PhaROS should be installed instead of Pharo. Note that there may be PhaROS installation issues²⁾. LRP should then be installed from source in the package of choice (see above). LRP requires the Cairo graphics library, which is may not be installed. To install, do a sudo apt-get install libcairo2:i386 in a terminal.

The ROS bridge is installed as follows. Note: both Gofer load directives are required.

Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  package: 'LiveRobotics-Bridge-PhaROS';
  load.
Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  package: 'LiveRobotics-UI-PhaROS';
  load

Next time the LRP interpreter is opened the ROS bridge UI will open, asking for the name of the class that represents the current package.

The Mindstorms EV3 can be remote-controlled over WiFi if the Lego-supported USB WiFi adapter is fitted (a "NETGEAR WiFi dongle WNA1100 Wireless-N 150"). Note: only this exact adapter is supported by default. Connecting to a WiFi network is relatively straightforward, and there are also tutorials on-line e.g. the "Connecting the EV3 to a network" section of this page.

First install JetStorm (instructions adapted from the JetStorm install page).

Gofer it
  smalltalkhubUser: 'JLaval' project: 'JetStorm';
  configuration;
  loadBleedingEdge.

Second install the JetStorm bridge as follows:

Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  package: 'LiveRobotics-Bridge-JetStorm';
  load

Next time the LRP interpreter is opened the Jetstorm bridge UI will open, asking for the IP address of the EV3 and then setting up the connection.

Note that the proprietary Lego protocol sadly does not fulfill usual Lego robustness standards and connection setup easily fails on a busy network. One way to see whether the EV3 is broadcasting its availability is by using the tcpdump command, as shown in the example below, for a brick that has an IP address of 192.168.0.87 :

Makivi:~ jfabry$ sudo tcpdump src 192.168.0.87 and udp
Password:
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pktap, link-type PKTAP (Packet Tap), capture size 65535 bytes
15:51:35.666729 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:40.582201 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:45.600016 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:50.617837 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:55.637489 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67

A message should be received at least every 5 seconds (as above) for the connection to be set up successfully. If you receive messages at a lower frequency, the network is probably too overloaded or the WiFi radio spectrum too noisy for the connection to work. Note that if you receive no packets at all, this may be because your base station filters out these packets.

For example code that uses the EV3 we refer to our GitHub repository.

The bug tracker for LRP is hosted by GitHub.