Confined Gradual Typing: Benchmarks

The benchmarks used in the paper “Confined Gradual Typing” can be downloaded here: OOPSLA paper benchmark suite. This file also include example images for SCGT and RCGT.

For instructions on how to use these benchmarks and examples, please continue reading this page or read the included readme.txt.

The benchmark suite is ready to run in a Windows or Linux machine. To run in another OS with bash, the corresponding Cog VM needs to be downloaded. The benchmarks have only been run on Windows and Linux. For MacOS, the path to the actual virtual machine inside the application package is CogVM.app/Contents/MacOS/CogVM Also on Mac, the VM does not run completely headless, please ignore the window that pops up when the benchmarks are run.

Contents of the image

This image includes:

  • examples of SCGT and RCGT
  • the micro and macro benchmarks

Also, the Windows Cog VM is included in the image. To run in an OS with bash, put the corresponding Cog VM in the “vm” directory and modify the $VM variable of the linux benchmarks scripts.

Examples

The example images of SCGT and RCGT are located in the example folder.

To run the SCGT image, execute: 

Win:   .\sggt.bat
Linux: ./sggt.sh

To run the RCGT image, execute: 

Win:   .\rggt.bat
Linux: ./rggt.sh

Qualifier syntax

The syntax for the qualifiers are: 

Up qualifier:   ^^   Ex: ^^Integer, ^^(->String)
Down qualifier: vv   Ex: vvString, vv(Integer->vvString)

Benchmarks

The micro and macro benchmark are configured to run in a machine with Windows or Linux.

Micro-benchmarks

To run the micro-benchmarks, go to the directory “benchmark/micro” and execute: 

Win:   .\bench.bat
Linux: ./bench.sh

The results of the microbenchmarks are stored in the results directory inside the directory of the microbenchmarks (“benchmark/micro/results”) in the file results-X.txt, where X is the semantics used:

  1. GT for non-gradual Gradual Typing
  2. RGGT for Relaxed Gradual Gradual Typing

In the results-X.txt file, each line states:

  1. the size of the collection.
  2. the time taken in:
    1. Wrapper Creation , NQ, Wrapped.
    2. Closure Evaluation, NQ, Wrapped.
    3. Wrapper Creation , NQ, Unwrapped.
    4. Closure Evaluation, NQ, Unwrapped.
    5. Wrapper Creation , NQ, NoDyn.
    6. Closure Evaluation, NQ, NoDyn.
    7. Wrapper Creation , DQ, Unwrapped (only RGGT).
    8. Closure Evaluation, DQ, Unwrapped (only RGGT).
    9. Wrapper Creation , DQ, NoDyn (only RGGT).
    10. Closure Evaluation, DQ, NoDyn (only RGGT).
    11. Wrapper Creation , UQ, Unwrapped (only RGGT).
    12. Closure Evaluation, UQ, Unwrapped (only RGGT).
    13. Wrapper Creation , UQ, NoDyn (only RGGT).
    14. Closure Evaluation, UQ, NoDyn (only RGGT).

Legend: NQ=No Qualifiers, DQ=Down arrow qualifier, UQ=Up arrow qualifier

Macro-benchmarks

To run the macro-benchmarks, go to the directory “benchmark/macro” and execute:

Win: .\bench.bat Linux: ./bench.sh

The results of the microbenchmarks are stored in the results directory inside the directory of the macrobenchmarks (“benchmark/macro/results”) in the file results-X.txt, where X is either:

  1. GT-Wrapped
  2. GT-Unwrapped
  3. RGGT-Wrapped
  4. RGGT-Wrapped-NQ
  5. RGGT-Wrapped-DQ
  6. RGGT-Wrapped-UQ

Legend: NQ=No Qualifiers, DQ=Down arrow qualifier, UQ=Up arrow qualifier

Each file containts the time taken to run the macro-benchmark in that variant.