Live Robot Programming
Live Robot Programming (LRP) is a live programming language designed for the creation of the behavior layer of robots. It is fundamentally a nested state machine language built with robotics applications in mind, but it is not bound to a specific robot middleware, API or OS.
We all like videos of robots, right? Here are some of LRP on a few different robots for your viewing pleasure.
- How does a PR2 move through a door with LRP? (This uses ROS to control the robot.)
- Flying a Parrot AR.Drone 2, using AR tags. (This uses the Parrot API to control the drone.)
- Making a NAO robot follow a ball. (This uses the NAO API to control the robot.)
- PR2 pick and place interaction with speech control. (This uses ROS to control the robot.)
- Journal paper defining LRP, the functioning of the interpreter and bridges to robot API accepted for Science of Computer Programming, a preprint is available.
- LRP was part of Johan's talk at the robotics day conference, claimed to be the largest robotics event in Latin America.
Have a look at one minute of LRP coding: making a state machine for the tick-tock of an oldfashioned analog watch:
As LRP is not bound to a specific robot infrastructure, bridges to a given infrastructure are add-ons to the language. The language can therefore also be used 'simply' as a nested state machine live programming language! Currently LRP provides a bridge to ROS, to the Lego Mindstorms EV3 via JetStorm and to the Parrot AR.Drone 2.
LRP is implemented in Pharo Smalltalk, so basic knowledge of Smalltalk is required to be able to write useful code. But don't panic, Pharo itself comes with a set of tutorials that are sufficient for being able to use LRP.
Overview of the Language Features
LRP programs describe nested state machines: a state of a machine may contain (several) complete state machine(s), whose states may again contain a machine, and so on. Next to states, machines may also define transitions and events. Events are named actions (= Smalltalk blocks) that specify the conditions for transitions. LRP has four kinds of transitions:
- normal transitions: the usual ones
- epsilon transitions: who trigger automatically when the state is active
- timeout transitions: who trigger after their timeout (in milliseconds) given as a literal or as a variable reference
- wildcard transitions: who have no specific source state, they consider all of the states in their machine as the source state.
The following is an example state machine (called
esc) that models the escapement of a mechanical clock (the part responsible for the tick-tock). It consists of two states (
tock) and the timeout transitions between them. To the right is its visual representation in the LRP editor.
(machine esc (state tick) (state tock) (ontime 500 tick -> tock) (ontime 500 tock -> tick) )
To start interpreting this machine, a
spawn statement needs to be specified. Spawns can be declared either at the top level, for the top level machine or as the
onentry action of a state (see below for
onentry), for nested machines. The video at the top of this page shows the process of writing and running this code.
(spawn esc tick)
States may also contain actions: an on entry action (
onentry), an on exit action (
onexit) and a running action (
running). Actions are Smalltalk blocks. When a state becomes active, its on entry action is executed once, completely. When the state stops being active, its on exit action is executed once, completely. While the state is active, its running action is executed as the main part of the interpretation loop. In other words, the block is executed repeatedly, and between each execution of the block the events are checked.
A machine can also define variables. Global variables may also be defined, outside of the root machine. Variables must be given a value when they are defined, the value is the result of evaluating a Smalltalk block. This block has in scope all variables that are lexically in scope. All actions (i.e.
running blocks) may read, send messages to, and set all variables in their lexical scope.
For example, to add a seconds counter to the escapement, the code is as below. We show a screenshot of the complete editor here. It shows the code pane (left), the tree of machines and current variables (center), and the visualization of the machine (right).
Another example is a bit more complicated: a resettable timer that shows minutes as a variable, and seconds (rounded to 10) as a state. To the right of the code we show the relevant part of the editor. The
go transition is an epsilon transition, the
reset transition is a wildcard transition because of the
* before the arrow replacing the source state.
;; a resettable timer with 10 sec intervals (var minute := ) (machine timer (state zero) (state ten) (state twenty) (state thirty) (state fourty) (state fifty (onexit [minute := minute + 1])) (ontime 10000 zero -> ten toten) (ontime 10000 ten -> twenty totwenty) (ontime 10000 twenty -> thirty tothirty) (ontime 10000 thirty -> fourty tofourty) (ontime 10000 fourty -> fifty tofifty) (ontime 10000 fifty -> zero tozero) (var doreset := ) (state init (onentry [minute := 0. doreset := 0])) (on resetting *-> init reset) (eps init -> zero go) (event resetting [doreset = 1]) ) (spawn timer zero)
Below is a video that shows this machine in action, and demonstrates that when the
doreset variable is set to 1
- the wildcard transition occurs to
- whose onentry action resets
minuteto 0 and
- which is then followed by the epsilon transition to
As a variation on the clock theme, below is the example for a stopwatch. The values of the variables influence whether the stopwatch is running or stopped, and if it should be reset. Also, the stopwatch state machine is an example that uses all transition types available in LRP.
;;; Stopwatch (machine stopwatch (var start := [false]) (var reset := [false]) (var seconds := ) (state waiting ) (on starting waiting -> tick) (event starting [start]) (state tick) (state tock (onexit [seconds := seconds + 1])) (ontime 500 tick -> tock ) (ontime 500 tock -> tick ) (on reset *-> resetting) (event reset [reset]) (state resetting (onentry [reset := false. seconds := 0] )) (eps resetting -> tick tet) (on stop *-> waiting tes) (event stop [start not]) ) (spawn stopwatch waiting)
As LRP is a nested state machine language, a state of a machine may contain (several) complete state machine(s), whose states may again contain a machine, and so on. Machines can be spawned when the state is entered by having the
onentry statement contain a
spawn statement instead of an action block. The machine specified in such a spawn can be any machine that is in lexical scope. This allows the machine to be spawned to be defined in the spawning state or any parent state, recursively up to the root (including other machines defined at root level).
onentryspawn, a machine that is contained lexically in the state is not executed.
When a state with a nested machine stops being active, the nested machine is stopped and discarded. As part of this process, the on exit action in the active state of this nested machine is performed after any machines nested in that state are stopped and discarded (and so on recursively).
A brief (contrived) example is as follows:
(machine root (state r1 (machine nest1 (state n1 (machine nest2 (state n1n2)) (onentry (spawn nest2 n1n2)) ) (ontime 500 n1->n2) (state n2 (machine nest3 (state n2n3)) (onentry (spawn nest3 n2n3)) ) (state n3) (ontime 500 n2->n3)) (onentry (spawn nest1 n1)) ) (state r2) (ontime 2000 r1->r2) (ontime 500 r2->r1) ) (spawn root r1)
It is possible to have multiple machines running at the same time, i.e. to have multiple machines each with an active state, without these machines being nested. To achieve this, multiple
spawn statements are specified, either at top level or as
onentry actions of a state. Concurrency obeys the following rules:
- If a spawn statement specifies a machine that is already running, i.e. that already has an active state, the spawn fails with an error.
- exiting a state that has multiple nested machines concurrently running means exiting all of these machines
- removing a top-level spawn does NOT stop the machine that was spawned
- adding or removing a spawn of a state means changing the state. So if the state is active (possibly having multiple nested machines running) the program is restarted.
The scheduling of machine execution is fair and predictable:
- all top level spawned machines perform one interpretation step for each interpreter step. This happens sequentially and the order is the lexical order of the spawn statements.
- all nested machines perform one interpretation step for each interpretation step of the state. This happens sequentially and the order is the lexical order of the spawn statements. Note that this implies recursion when an active state of a nested machine has concurrent nested state machines.
Put succinctly, interpretation of multiple running machines is equal to a depth-first traversal of the running machines tree.
There are two brief (contrived) examples:
(machine m (state s) (state t) (ontime 1500 s->t) (ontime 1500 t->s) ) (machine n (state a) (state b) (state c) (ontime 2000 a->b) (ontime 2000 b->c) (ontime 2000 c->a) ) (spawn m s) (spawn n a)
(machine m (state s) (state t) (ontime 1500 s->t) (ontime 1500 t->s) ) (machine n (state a) (state b) (state c) (ontime 2000 a->b) (ontime 2000 b->c) (ontime 2000 c->a) ) (machine o (state p (onentry (spawn m s)) (onentry (spawn n a))) ) (spawn o p)
In a nested machine it is possible to define transitions that go to a state of the parent machine, effectively exiting the nested machine. Such transitions are like normal transactions, except that their keyword is
exit and the destination state should be a state of the parent machine. Note that exit transitions are in fact syntactic sugar.
A simple example is as follows. As soon as the
out variable is set to
true, the nested machine is exit.
(machine root (var out := [false]) (state one (machine nested (state onen) (exit goout onen->two) (event goout [out])) (onentry (spawn nested onen)) ) (state two) ) (spawn root one)
It can become tedious for transitions to need an event as a trigger, since it requires the definition of an event as a separate statement. This is especially tedious when the transition is the only that references that event. To ease this tedium, transitions also accept a block instead of an event name. This block should return true for the transition to trigger.
Eventless transitions are in fact syntactic sugar: an event is generated and added to the machine, with as action block the block that was specified in the transition, and the transition instead then refers to that event.
User interface: Transition to and Jump to
The LRP user interface allows for the user to force a machine in a given state. By right-clicking on a state in the visualisation a menu appears, with the option to
transition to or
jump to. The former acts as if a transition is added from the currently active state to the selected state, and this transition is removed immediately after it is taken. The latter also transitions to the given state, however without running the
onentry actions of all affected states. (Recall that if a state has a nested machine, its active state
onexit actions are normally also executed.)
Transition to and jump to also combine with concurrency (see above for concurrency): if this machine and none of its parents is running, the machine is spawned as a top-level spawn. If it (or its parents) is running, the active state of that running machine is considered as the state from which the transition or jump starts.
LRP is available on SmalltalkHub, under the MIT license. To be able to use it, you first need to download Pharo 5 and basic knowledge of Pharo is required. Note that Pharo itself comes with the tutorials required to get you up to speed.
You can download a working image of the latest development version from our continuous integration server. This image is ready to work, containing LRP and any other project that LRP needs. In this image you will have a new menu entry in the World menu that opens the LRP UI. (Click on the background to get the World menu.)
Installing LRP From Source
Alternatively, you can install LRP from source, do-it of the following in a playground:
Gofer it smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming'; configuration; loadDevelopment
ROS integration is provided thanks to PhaROS. You can download a working image with LRP and PhaROS that is relatively up-to-date. This image is ready to work, you will have a new menu entry in the World menu that opens the LRP UI. (Click on the background to get the World menu.)
When the interpreter is opened, a separate UI allows for making subscriptions to topics and declaring topics on which to publish. For example code that uses ROS we refer to our GitHub repository.
We will assume you are using a Ubuntu linux, as this is the standard OS for ROS. To install from source, first PhaROS should be installed instead of Pharo. Note that there may be PhaROS installation issues1). LRP should then be installed from source in the package of choice (see above). LRP requires the Cairo graphics library, which is may not be installed. To install, do a
sudo apt-get install libcairo2:i386 in a terminal.
The ROS bridge is installed as follows. Note: both Gofer load directives are required.
Gofer it smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming'; package: 'LiveRobotics-Bridge-PhaROS'; load. Gofer it smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming'; package: 'LiveRobotics-UI-PhaROS'; load
Next time the LRP interpreter is opened the ROS bridge UI will open, asking for the name of the class that represents the current package.
Parrot AR.Drone Support
The Parrot AR.Drone 2 is also supported in LRP. To use it, get the Pharo API from http://smalltalkhub.com/#!/~CaroHernandez/ArDronePharo, the documentation of the API is also at this repository and the LRP bridge can also be found there.
Example LRP code for the drone is available on GitHub.
Lego Mindstorms EV3 Support
The Mindstorms EV3 can be remote-controlled over WiFi if the Lego-supported USB WiFi adapter is fitted (a "NETGEAR WiFi dongle WNA1100 Wireless-N 150"). Note: only this exact adapter is supported by default. Connecting to a WiFi network is relatively straightforward, and there are also tutorials on-line e.g. the "Connecting the EV3 to a network" section of this page.
First install JetStorm (instructions adapted from the JetStorm install page).
Gofer it smalltalkhubUser: 'JLaval' project: 'JetStorm'; configuration; loadBleedingEdge.
Second install the JetStorm bridge as follows:
Gofer it smalltalkhubUser: 'jfabry' project: 'LiveRobotProgramming'; package: 'LiveRobotics-Bridge-JetStorm'; load
Next time the LRP interpreter is opened the Jetstorm bridge UI will open, asking for the IP address of the EV3 and then setting up the connection.
Note that the proprietary Lego protocol sadly does not fulfill usual Lego robustness standards and connection setup easily fails on a busy network. One way to see whether the EV3 is broadcasting its availability is by using the tcpdump command, as shown in the example below, for a brick that has an IP address of 192.168.0.87 :
Makivi:~ jfabry$ sudo tcpdump src 192.168.0.87 and udp Password: tcpdump: data link type PKTAP tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on pktap, link-type PKTAP (Packet Tap), capture size 65535 bytes 15:51:35.666729 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67 15:51:40.582201 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67 15:51:45.600016 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67 15:51:50.617837 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67 15:51:55.637489 IP 192.168.0.87.54772 > 192.168.0.255.nati-dstp: UDP, length 67
A message should be received at least every 5 seconds (as above) for the connection to be set up successfully. If you receive messages at a lower frequency, the network is probably too overloaded or the WiFi radio spectrum too noisy for the connection to work. Note that if you receive no packets at all, this may be because your base station filters out these packets.
For example code that uses the EV3 we refer to our GitHub repository.
The bug tracker for LRP is hosted by GitHub.
sudo apt-get install libX11-6:i386 libxext6:i386 lib32z1 libasound2:i386 libasound2-plugins:i386 libssl0.9.8:i386 libfreetype6:i386then (for Indigo)
source /opt/ros/indigo/setup.bashthen an alternate installation
curl http://car.mines-douai.fr/scripts/PhaROS | sed 's/libgl1-mesa-glx:i386/ /' | bash