Live Robot Programming

Live Robot Programming (LRP) is a live programming language designed for the creation of the behavior layer of robots. It is fundamentally a nested state machine language built with robotics applications in mind, but it is not bound to a specific robot middleware, API or OS.

LRP is open source, download instructions are below, example code can be found online and there is a public bug tracker.

We all like videos of robots, right? Here are some of LRP on a few different robots for your viewing pleasure.

• Moving the Duckiebot platform using LRP (This uses ROS to control the robot.)
• How does a PR2 move through a door with LRP? (This uses ROS to control the robot.)
• Flying a Parrot AR.Drone 2, using AR tags. (This uses the Parrot API to control the drone.)
• Making a NAO robot follow a ball. (This uses the NAO API to control the robot.)
• PR2 pick and place interaction with speech control. (This uses ROS to control the robot.)

NEWS:

• Miguel presented LRP at the Cochabamba Summer School 2017, at the Universidad Mayor the San Simón, Cochabamba, Bolivia.
• Parrot AR.Drone 2 support is finished!
• A first version of NAO support is out!
• Journal paper defining LRP, the functioning of the interpreter and bridges to robot API accepted for Science of Computer Programming, a preprint is available.
• NIER Paper on visualizing robotic sensors accepted at VISSOFT 2015, a preprint is available.
• Johan presented LRP at the ESUG 2015 conference. A video of the talk is available as part of the conference playlist on Youtube.
• LRP was part of Johan's talk at the robotics day conference, claimed to be the largest robotics event in Latin America.

Have a look at one minute of LRP coding: making a state machine for the tick-tock of an oldfashioned analog watch:

As LRP is not bound to a specific robot infrastructure, bridges to a given infrastructure are add-ons to the language. The language can therefore also be used 'simply' as a nested state machine live programming language! Currently LRP provides a bridge to ROS, to the Lego Mindstorms EV3 via JetStorm and to the Parrot AR.Drone 2.

LRP is implemented in Pharo Smalltalk, so basic knowledge of Smalltalk is required to be able to write useful code. But don't panic, Pharo itself comes with a set of tutorials that are sufficient for being able to use LRP.

LRP programs describe nested state machines: a state of a machine may contain (several) complete state machine(s), whose states may again contain a machine, and so on. Next to states, machines may also define transitions and events. Events are named actions (= Smalltalk blocks) that specify the conditions for transitions. LRP has four kinds of transitions:

• normal transitions: the usual ones
• epsilon transitions: who trigger automatically when the state is active
• timeout transitions: who trigger after their timeout (in milliseconds) given as a literal or as a variable reference
• wildcard transitions: who have no specific source state, they consider all of the states in their machine as the source state.

The following is an example state machine (called esc) that models the escapement of a mechanical clock (the part responsible for the tick-tock). It consists of two states (tick and tock) and the timeout transitions between them. To the right is its visual representation in the LRP editor.

(machine esc
	(state tick)
	(state tock)
	(ontime 500 tick -> tock)
	(ontime 500 tock -> tick)
	)

To start interpreting this machine, a spawn statement needs to be specified. Spawns can be declared either at the top level, for the top level machine or as the onentry action of a state (see below for onentry), for nested machines. The video at the top of this page shows the process of writing and running this code.

(spawn esc tick)

States may also contain actions: an on entry action (onentry), an on exit action (onexit) and a running action (running). Actions are Smalltalk blocks. When a state becomes active, its on entry action is executed once, completely. When the state stops being active, its on exit action is executed once, completely. While the state is active, its running action is executed as the main part of the interpretation loop. In other words, the block is executed repeatedly, and between each execution of the block the events are checked.

A machine can also define variables. Global variables may also be defined, outside of the root machine. Variables must be given a value when they are defined, the value is the result of evaluating a Smalltalk block. This block has in scope all variables that are lexically in scope. All actions (i.e. onentry, onexit, running blocks) may read, send messages to, and set all variables in their lexical scope.

For example, to add a seconds counter to the escapement, the code is as below. We show a screenshot of the complete editor here. It shows the code pane (left), the tree of machines and current variables (center), and the visualization of the machine (right).

Another example is a bit more complicated: a resettable timer that shows minutes as a variable, and seconds (rounded to 10) as a state. To the right of the code we show the relevant part of the editor. The go transition is an epsilon transition, the reset transition is a wildcard transition because of the * before the arrow replacing the source state.

;; a resettable timer with 10 sec intervals
(var minute := [0])
(machine timer
  (state zero)
  (state ten)
  (state twenty)
  (state thirty)
  (state fourty)
  (state fifty
    (onexit [minute := minute + 1]))
  (ontime 10000 zero -> ten toten)
  (ontime 10000 ten -> twenty totwenty)
  (ontime 10000 twenty -> thirty tothirty)
  (ontime 10000 thirty -> fourty tofourty)
  (ontime 10000 fourty -> fifty tofifty)
  (ontime 10000 fifty -> zero tozero)

  (var doreset := [0])
  (state init
    (onentry [minute := 0. doreset := 0]))
  (on resetting *-> init reset)
  (eps init -> zero go)
  (event resetting [doreset = 1])
)
(spawn timer zero)

Below is a video that shows this machine in action, and demonstrates that when the doreset variable is set to 1

• the wildcard transition occurs to init,
• whose onentry action resets minute to 0 and doreset to 0,
• which is then followed by the epsilon transition to zero.

As a variation on the clock theme, below is the example for a stopwatch. The values of the variables influence whether the stopwatch is running or stopped, and if it should be reset. Also, the stopwatch state machine is an example that uses all transition types available in LRP.

;;; Stopwatch
(machine stopwatch
  (var start := [false]) (var reset := [false]) (var seconds := [0])
  (state waiting )
  (on starting waiting -> tick)
  (event starting [start])
  (state tick)
  (state tock (onexit [seconds := seconds + 1]))
  (ontime 500 tick -> tock )
  (ontime 500 tock -> tick )

  (on reset *-> resetting)
  (event reset [reset])
  (state resetting (onentry [reset := false. seconds := 0] ))
  (eps resetting -> tick tet)

  (on stop *-> waiting tes)
  (event stop [start not])
)
(spawn stopwatch waiting)

As LRP is a nested state machine language, a state of a machine may contain (several) complete state machine(s), whose states may again contain a machine, and so on. Machines can be spawned when the state is entered by having the onentry statement contain a spawn statement instead of an action block. The machine specified in such a spawn can be any machine that is in lexical scope. This allows the machine to be spawned to be defined in the spawning state or any parent state, recursively up to the root (including other machines defined at root level).

When a state with a nested machine stops being active, the nested machine is stopped and discarded. As part of this process, the on exit action in the active state of this nested machine is performed after any machines nested in that state are stopped and discarded (and so on recursively).

A brief (contrived) example is as follows:

(machine root
	(state r1
		(machine nest1 
			(state n1
				(machine nest2 (state n1n2))
				(onentry (spawn nest2 n1n2))
				)
			(ontime 500 n1->n2)
			(state n2
				(machine nest3 (state n2n3))
				(onentry (spawn nest3 n2n3))
				)
			(state n3)
			(ontime 500 n2->n3))
		(onentry (spawn nest1 n1))
		)
	(state r2)
	(ontime 2000 r1->r2)
	(ontime 500 r2->r1)
)
(spawn root r1)

It is possible to have multiple machines running at the same time, i.e. to have multiple machines each with an active state, without these machines being nested. To achieve this, multiple spawn statements are specified, either at top level or as onentry actions of a state. Concurrency obeys the following rules:

• If a spawn statement specifies a machine that is already running, i.e. that already has an active state, the spawn fails with an error.
• exiting a state that has multiple nested machines concurrently running means exiting all of these machines
• removing a top-level spawn does NOT stop the machine that was spawned
• adding or removing a spawn of a state means changing the state. So if the state is active (possibly having multiple nested machines running) the program is restarted.

The scheduling of machine execution is fair and predictable:

• all top level spawned machines perform one interpretation step for each interpreter step. This happens sequentially and the order is the lexical order of the spawn statements.
• all nested machines perform one interpretation step for each interpretation step of the state. This happens sequentially and the order is the lexical order of the spawn statements. Note that this implies recursion when an active state of a nested machine has concurrent nested state machines.

Put succinctly, interpretation of multiple running machines is equal to a depth-first traversal of the running machines tree.

There are two brief (contrived) examples:

(machine m
	(state s) (state t)
	(ontime 1500 s->t) (ontime 1500 t->s)
)
(machine n
	(state a) (state b) (state c)
	(ontime 2000 a->b) (ontime 2000 b->c) (ontime 2000 c->a)
)
(spawn m s)
(spawn n a)

(machine m
	(state s) (state t)
	(ontime 1500 s->t) (ontime 1500 t->s)
)
(machine n
	(state a) (state b) (state c)
	(ontime 2000 a->b) (ontime 2000 b->c) (ontime 2000 c->a)
)
(machine o
	(state p
		(onentry (spawn m s))
		(onentry (spawn n a)))
)
(spawn o p)

In a nested machine it is possible to define transitions that go to a state of the parent machine, effectively exiting the nested machine. Such transitions are like normal transactions, except that their keyword is exit and the destination state should be a state of the parent machine. Note that exit transitions are in fact syntactic sugar.

A simple example is as follows. As soon as the out variable is set to true, the nested machine is exit.

(machine root
	(var out := [false])
	(state one
		(machine nested
			(state onen)
			(exit goout onen->two)
			(event goout [out]))
		(onentry (spawn nested onen))
	)
	(state two)
)
(spawn root one)

It can become tedious for transitions to need an event as a trigger, since it requires the definition of an event as a separate statement. This is especially tedious when the transition is the only that references that event. To ease this tedium, transitions also accept a block instead of an event name. This block should return true for the transition to trigger.

Eventless transitions are in fact syntactic sugar: an event is generated and added to the machine, with as action block the block that was specified in the transition, and the transition instead then refers to that event.

The LRP user interface allows for the user to force a machine in a given state. By right-clicking on a state in the visualisation a menu appears, with the option to transition to or jump to. The former acts as if a transition is added from the currently active state to the selected state, and this transition is removed immediately after it is taken. The latter also transitions to the given state, however without running the onexit and onentry actions of all affected states. (Recall that if a state has a nested machine, its active state onexit actions are normally also executed.)

Transition to and jump to also combine with concurrency (see above for concurrency): if this machine and none of its parents is running, the machine is spawned as a top-level spawn. If it (or its parents) is running, the active state of that running machine is considered as the state from which the transition or jump starts.

LRP is available on SmalltalkHub, under the MIT license. To be able to use it, you first need to download Pharo 5 and basic knowledge of Pharo is required. Note that Pharo itself comes with the tutorials required to get you up to speed.

You can download a working image of the latest development version from our continuous integration server. This image is ready to work, containing LRP and any other project that LRP needs. In this image you will have a new menu entry in the World menu that opens the LRP UI. (Click on the background to get the World menu.)

We have example code online in our GitHub repository. To work with ROS or the Lego Mindstorms EV3 via JetStorm, download the respective bridge packages, as detailed below.

Installing LRP From Source

Alternatively, you can install LRP from source, do-it of the following in a playground:

Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  configuration;
  loadDevelopment

ROS integration is provided thanks to PhaROS. You can download a working image with LRP and PhaROS that is relatively up-to-date. This image is ready to work, you will have a new menu entry in the World menu that opens the LRP UI. (Click on the background to get the World menu.)

When the interpreter is opened, a separate UI allows for making subscriptions to topics and declaring topics on which to publish. For example code that uses ROS we refer to our GitHub repository.

From Source

We will assume you are using a Ubuntu linux, as this is the standard OS for ROS. To install from source, first PhaROS should be installed instead of Pharo. Note that there may be PhaROS installation issues¹⁾. LRP should then be installed from source in the package of choice (see above). LRP requires the Cairo graphics library, which is may not be installed. To install, do a sudo apt-get install libcairo2:i386 in a terminal.

The ROS bridge is installed as follows. Note: both Gofer load directives are required.

Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  package: 'LiveRobotics-Bridge-PhaROS';
  load.
Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  package: 'LiveRobotics-UI-PhaROS';
  load

Next time the LRP interpreter is opened the ROS bridge UI will open, asking for the name of the class that represents the current package.

If you have problems installing PhaROS, you can bypass the main installation by downloading only the PhaROS API for Pharo.

Gofer it
  smalltalkhubUser: 'CAR' project: 'PhaROS';
  configuration;
  load

After installing PhaROS, you can install the LRP ROS bridge (see above).

The Parrot AR.Drone 2 is also supported in LRP. To use it, get the Pharo API from http://smalltalkhub.com/#!/~CaroHernandez/ArDronePharo, the documentation of the API is also at this repository and the LRP bridge can also be found there.

Example LRP code for the drone is available on GitHub.

The Mindstorms EV3 can be remote-controlled over WiFi if the Lego-supported USB WiFi adapter is fitted (a "NETGEAR WiFi dongle WNA1100 Wireless-N 150"). Note: only this exact adapter is supported by default. Connecting to a WiFi network is relatively straightforward, and there are also tutorials on-line e.g. the "Connecting the EV3 to a network" section of this page.

First install JetStorm (instructions adapted from the JetStorm install page).

Gofer it
  smalltalkhubUser: 'JLaval' project: 'JetStorm';
  configuration;
  loadBleedingEdge.

Second install the JetStorm bridge as follows:

Gofer it
  smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming';
  package: 'LiveRobotics-Bridge-JetStorm';
  load

Next time the LRP interpreter is opened the Jetstorm bridge UI will open, asking for the IP address of the EV3 and then setting up the connection.

Note that the proprietary Lego protocol sadly does not fulfill usual Lego robustness standards and connection setup easily fails on a busy network. One way to see whether the EV3 is broadcasting its availability is by using the tcpdump command, as shown in the example below, for a brick that has an IP address of 192.168.0.87 :

Makivi:~ jfabry$ sudo tcpdump src 192.168.0.87 and udp
Password:
tcpdump: data link type PKTAP
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on pktap, link-type PKTAP (Packet Tap), capture size 65535 bytes
15:51:35.666729 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:40.582201 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:45.600016 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:50.617837 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
15:51:55.637489 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67

A message should be received at least every 5 seconds (as above) for the connection to be set up successfully. If you receive messages at a lower frequency, the network is probably too overloaded or the WiFi radio spectrum too noisy for the connection to work. Note that if you receive no packets at all, this may be because your base station filters out these packets.

For example code that uses the EV3 we refer to our GitHub repository.

The bug tracker for LRP is hosted by GitHub.