General documentation

This document refers to VPL 3.X and execution(jail) server 2.X

Security of an execution server

Safety is a primary goal in the design of VPL. All execution or testing of programs of students or professors in VPL are carried out in an execution server. The VPL module in the Moodle server only manages data, it does not execute any code outside himself.

The first approach to minimize the effects of any type of attack has been to separate the data handling system (the Moodle server) and the systems that execute code (the execution servers). With this distribution of work we got that VPL don't has more risks than any other Moodle module that don't run programs. The execution servers can be configured to be stateless systems. They can do their job and don't be affected by them. Due to the impossibility to ensure that a system is immune to all kinds of attack it is recommended that the execution servers be hosted in virtual machines. This allow to easily reset the execution server to a known state.

The VPL execution service is responsible for receiving and controlling the execution of code. The execution can be terminated for four reasons:

  1. Finish their execution normally
  2. They are stopped when they deplete their assigned resources (time, memory, etc.)
  3. They are stopped on user request (e.g. the user closes the browser). This monitoring is done through a websocket connection from the browser to the execution server.
  4. The Moodle server requests the stop of the task. A user can only have one running task, if the user requests to execute another task, the previous one will be stopped.

After the end of a task, the work area used is cleaned.

The VPL module uses http+XMLRPC to communicate with the execution server. The browser uses WebSocket (ws:) for monitoring the task and the interactive execution. It is also possible to use https and wss (secure connection). If communications from the browser with the Moodle server are using https most browsers require the use of wss to connect with the execution server. Using https and wss required to have a certificate in the execution server. By default, the VPL installer generate a self-signed certificate which may generate problems for some browsers like (IExplorer and Safari) To prevent the request to users accepting self-signed certificates, you must use certificates signed by a known Certificate Authority.

Creating a simple activity

Follow these steps:

 

  1. Create a VPL activity
  2. Set the name of the activity and its description
  3. Set the deadline
  4. Set the maximum number of files that every student can upload
  5. Set the maximum score to get
  6. Save the new activity
  7. Go to options and set the execution actions (run/debug/evaluate) that the student can do
  8. It is desirable to set, in "Required Files", the names of the files that the student must submit.

Connection generated when a execution is requested

  • Browser === AJAX (json) === > Moodle Server
  • Moodle Server === http/https (XMLRPC) === > Execution Server
  • Browser === ws/wss === > Execution Server

Process of a student's program execution (summarized)

The following steps are performed:

  1. Files submitted by the student are taken
  2. Files set by the teacher in "Execution files" are taken (this can replace files of student with same name)
  3. Depending on the action (run, debug or evaluate) the script file set by the teacher are taken. If there is no script, the system take a default script by detecting the programming language used based on the extension of filenames.
  4. In the case of evaluation, if there is no script, the VPL program for automatic assessment is also added. This program is based on the input and output of the program and requires that you specify the test cases in the file "vpl_evaluate.cases".
  5. This collected files are sent to a execution server
  6. The módule informs the browser that the execution has begun
  7. If the request is an evaluation, when the task is finished, the result of the evaluation is retrieved from the execution server.

Creating automated tests

To use the features of automatic evaluation of VPL programs must populate the "vpl_evaluate.cases" file.

This file has the following format :

"Case= case name": Sets the start and the name of a test case.

example:

Case = Positive Numbers

"input = text" can span multiple lines. It ends when another statement is entered.

example:

input = 3

4

"output = text " can span multiple lines . It ends with another statement. A test case can have multiple outputs statement. There are a few types of outputs: only numbers, text , exact text and regular expressions (*):

Numbers: only numbers are written. Only numbers are checked out, the rest of the text is ignored. The real numbers are checked with tolerance

example:

output = 4

5 6.8

Text: Only words are checked, the comparison is case-insensitive and the other characters are ignored.

example:

output = lemon orange

apple

Exact text : The text is enclosed in double quotes.

example:

output = "lemon orange

apple "

Regular Expressions (*): It begin with / and ends with / and optionally one or several modifiers. Format is similar to JavaScript literal, but POSIX regex is used.

example:

output = /lemon|orange|apple/i

Grade calculus

When an error (the output of a test case does not match with any expected value) is found, the value "grade_range/number_of_cases" is deducted from the mark. This value can be set by the teacher with the following statement:

"grade reduction = [ value | percent% ]". The discount can be a percentage or specific value, in other cases is used the general criterion.

(*) From VPL 3.1

How to measure the required resources for an execution in an execution(jail) server?

In general is correct to leave the defaults value of resource limits. If you want to restrict or extending the resources used  in a particular activity, it is recommended testing a standard solution of the activity, varying the resource from higher to lower to find the correct value.

Checking the installation of an execution server

If an execution server is operating properly it must respond with a page with the text OK if the following URLs are queried :

http://server_name[:port ]/OK

https://server_name[:secure ports]/OK

Add the following URL to the server list in the general module configuration

http://server_name [ : port ] / urlPath

(remember that urlPath is a parameter that is set in the configuration file of execution server "/etc/vpl/vpl-jail-system.conf " )

and in a VPL activity go to "administration VPL -> advanced settings -> Check execution servers " and verify that the Moodle server connects to the execution server.

Activities in person

VPL has several mechanisms to significantly increase the reliability of the authorship of in person activities. These mechanisms are mainly three: control access to activity by password, allowing access to IP machines or networks and limiting the write of code using the keyboard only. These mechanisms can be used in the combination that is deemed most convenient. All these mechanisms are set in the main settings of the activity.

Control Access by Password

You can set a password to each activity. The student needs to enter the password to access any element of that activity (except the title). Entering the password gives permission to access the activity during the current session of the student, even if the key is changed . A common use is to give students the password to start the activity and change it once introduced, which prevents to use the given password after the start of the activity.

Limiting access by IP

You can set which machines or networks can access the activity. This will prevent to access the activity from outside of the authorized networks . The format is

 

  • xxx.xxx.xxx.xxx /nn number of bits in the mask)
  • xxx.xxx.xxx.xxx yyy - IP range in the last group
  • xxx.xxx.xxx or xxx.xxx.xxx . incomplete address

 

It may be used multiple networks or IP separated by ","

Write code via keyboard

If you enable the "Only deliveries welcome from the restricted editor" the following is locked:

 

  • Upload files to be submitted.
  • Import files to the editor.
  • Paste or drag text from outside the editor, although it is possible to cut and paste within the editor.

Names and initial content of files to be submitted

You can create activities in which the students must complete a task. To do this, go to option and set the "required files" filenames and initial content you want to give. The user has the option to download the initial files from the description page. The first time a student use the editor it is initialized with the files and content established.

Creating scripts

The scripts for running/debugging and evaluation is a powerful tool for the creation of activities tailored to specific needs and using automated test mechanism different from the default. A simple way to start a own script is to take the default script for the language used as the start point. The default scripts are located in "/mod/vpl/jail/default_scripts/".

The purpose of the script is to compile or prepare the implementation of the action requested run/debug/evaluate. If the script succeed it will generate an executable file called vpl_execution to be run on a textual console or vpl_wexecution for execution in a graphics console.

Automatic evaluation

When an automatic evaluation is done, the output (standard output) of the execution is processed to obtain the comments and proposed grade.

The comment format is as follows:

A comment line would be a line starting with ' Comment: = >>' . A block comment would be contained between a line containing only '< | -' and other with '- | >' . The proposed grade is taken from the last line that begins with ' Grade: = >>' .

If the automatic assessment is set then the proposed grade becomes the final.

Text format of the assessment

The comments in the correction of an activity may have a better file format that allows the evaluation. The format is as follows :

Lines beginning with "-" are titles.

Lines starting with "> " are preformatted text "<pre>" .

The rest of the lines are considered content related to the previous title.

Expressions of the form filename : number generate hyperlink to the corresponding line of the file.

At the end of each title line, optionally, you can add a negative discount. This value represents the discount attributed the associated comment. This discount is never displayed to the students. This discount is used for the "calculate" button. Example:

- Error: infinite loop (-10)

Using binary files

Currently VPL don't supports the handling of binary files, but it is possible to overcome this limitation using files in base-64 format . Files with extension b64 are decoded and extension removed. Example: photo.jpg.b64 file in base64 becomes photo.jpg in binary.

Files in subdirectories

Both the VPL module as the execution server supports storing file in directories, but for now only the editor allow to specify the directory. The way  to indicate the directory is writing the relative path name as file name. For example, you can write as filename "src/lib/util.c" the util.c file will be located in the "src/lib" subdirectory of the user's home directory.

Activity testing

To test the activity you can switch temporarily the user role to student. But this way to test the activity will allow students to access the activity. However VPL allows test activities without changing the role by accessing the menu "Settings - > Test activity". There the teacher can access the activity in a similar way as a student does but in a specific workspace for him. The difference are the "list of previous submissions" and that teacher can use all options (run/debug/rate) regardless of the provisions in "options of execution"

"Based on" other activity

The option " based on " enables to develop generic activities that can be used as a basis for others. In many cases the generic activities are not used as activities to be used by students, their purpose is to establish a common framework for other activities. The effect of setting a "based on" activity are the following:

 

  • Execution files are taken. Predefined script files ( vpl_run.sh , vpl_debug.sh and vpl_evalaute.sh ) are concatenated (based on first).
  • Limits of resources. The activities have higher priority than the inherited value .
  • Variations, which are concatenated to generate multiple variations .
  • Maximum size of each file to upload
  • Description of the activity are concatenated (based on first) .

 

Difference of VPL 3 and earlier versions

The main differences of VPL 3 relative to earlier versions are two: the replacement of the Java applet and changing connection architecture with execution servers.

The Java Applet technology has been replaced by HTML5+Ajax+WebSocket allowing eliminate the requirement to have a Java pluging in the browser, and avoid frequent security messages while using this technology.

Eliminating the Java Applet has allowed to change the way the program execution is performed. It has changed from synchronous to asynchronous executions. This has decreasing dramatically the load on the Moodle server. Other important effect of this change is that the installation of VPL on a Moodle server does not requires opening any ports and is compatible with server clusters.

IMPORTANT: VPL 3.X requires that the browser connects to the execution/jail server to monitor the process. VPL 2.X kept a connection from the Moodle server to the execution server with a high resource use.

In VPL 3 all the execution servers must be reachable from internet. Only the service ports (http/ws and https/wss) of the servers need to be reachable.