# Manual for Running our Experiments

In this README, we explain how to use our code to reproduce the results of our experiments.

## Prerequisites

To simplify setup of the experiments, we also provide a Dockerfile.

### Setup with Docker

Simply execute

```
docker build -f ./Dockerfile -t cuemin-experiments:latest .
```

to create a Dockerimage with all necessary setup for the experiments.

### Manual Setup without Docker

We strongly recommend using Docker to run the experiments. If you want to setup the environment yourself for any reason, you will have to install a lot of dependencies to be able to run all experiments with all competitors.

For running RAS, you will need to set up CUDA 11.2.2 together with the cudnn8 library.
To use all options we provide in our experiment scripts, you will also need a Java JRE. We used OpenJDK 11.

To run the experiments and our code in general, you need a Python 3.10 environment. We recommend to use Anaconda to set up a virtual environment. If you have "conda" and "make" installed, you can simply run

```
conda create -n cuemin-venv python=3.10
```

to set up a virtual environment.

Make sure that Cython is available and install all the requirements:

```
pip install Cython && pip install -r requirements.txt
```

Some of our sources must be compiled with Cython:

```
make cython
```

Finally, set the PYTHONPATH environment variable to the source code directory, such that
our relative imports work.

On Linux:

```
export PYTHONPATH=.
```

On Windows:

```
set PYTHONPATH=.
```

## Running the Experiments on Synthetic Data

The basic command to run the experiment on synthetic data is

```
docker run --rm -it --memory=48g --entrypoint /bin/bash --volume ~/cuemin_results:/opt/results cuemin-experiments:latest -c 'python ${PYTHONPATH}/experiments/evaluation/run_reconstruction_single_queue_experiment.py  --A D1 D3 D5 Geo0.1 Geo0.2 Geo0.5 P1 P2 P5 P10 --BA D1 --BS D1 --D FCFS LCFS --S Geo0.2 Geo0.5 Geo0.1 Geo0.05 Geo0.01 Geo0.001 P10 P20 P50 P100 D5 D25 D50 D75 D100 NB-10-0.5 NB-30-0.5 NB-10-0.1 --c 1:30 --seed 42 --experiment_name fcfs_vs_lcfs_no_batching --down_sample_models_per_nr_of_servers --nr_of_categorical_features 0 --nr_of_numerical_features 0 --execution P32 --nr_of_jobs_per_server 10 --max_nr_of_models 1000 --max_nr_of_arriving_jobs 3000 && bash /scripts/plot_results_of_synthetic_experiments_docker.sh'
```

You will find the results under "~/cuemin_results". Feel free to play around with the parameters:

| Parameter                  | Description                                                       |
| -------------------------- | -----------------------------------------------------             |
| A                          | inter-arrivaltime distributions                                   |
| BA                         | distributions for batch arrival size                              |
| D                          | waiting areas in ground-truth models: FCFS LCFS PQc-FCFS PQc-LCFS |
| S                          | service time distributions in ground-truth models                 |
| BS                         | distribution of batch service size in ground-truth models         |
| c                          | range of number of servers, e.g. 1:30 in ground-truth models      |
| seed                       | sets seeds for random number generators                           |
| experiment_name            | prefix of the result folder                                       |
| down_sample_models_per_nr_of_servers | if set, balances number of models per number of servers |
| nr_of_jobs_per_server      | n/c, controls amount of available training data for rediscovery   |
| execution                  | S for single-threaded execution (not recommended), Px to run x ground-truth models in parallel. |
| max_nr_of_models           | maximum number of ground-truth models to generate                 |
| max_nr_of_arriving_jobs    | limits the number of generated jobs for arrival                   |
| nr_of_categorical_features | nr of categorical features to generate                            |
| nr_of_numerical_features   | nr_of_numerical_features                                          |

The meaning of the distributions is:
- D1 is a degenerate distribution with value 1
- P3 is a Poisson distribution with expected value 3
- Geo0.1 is a geometric distribution with probability parameter 0.1
- NB-10-0.1 is a negative binomial distribution with parameters 10 and 0,1

If you do not use Docker to run the experiments on synthetic data, the basic command is

```
python experiments/evaluation/run_reconstruction_single_queue_experiment.py  --A D1 D3 D5 Geo0.1 Geo0.2 Geo0.5 P1 P2 P5 P10 --BA D1 --BS D1 --D FCFS LCFS --S Geo0.2 Geo0.5 Geo0.1 Geo0.05 Geo0.01 Geo0.001 P10 P20 P50 P100 D5 D25 D50 D75 D100 NB-10-0.5 NB-30-0.5 NB-10-0.1 --c 1:30 --seed 42 --experiment_name fcfs_vs_lcfs_no_batching --down_sample_models_per_nr_of_servers --nr_of_categorical_features 0 --nr_of_numerical_features 0 --execution P32 --nr_of_jobs_per_server 10 --max_nr_of_models 1000 --max_nr_of_arriving_jobs 3000
```

To generate plots, you have to run "python experiments/evaluation/create_plots_single_queue_experiment.py". The script expects a parameter "--experiments" with the paths to the created result folders of the experiments.

We provide scripts to run the same set of experiments we ran for our paper:

```
docker run --rm -it --memory=48g --entrypoint /bin/bash --volume ~/cuemin_results:/opt/results cuemin-experiments:latest -c 'bash /scripts/synthetic_fcfs_lcfs_no_batching.sh && bash /scripts/plot_results_of_synthetic_experiments_docker.sh'
```

```
docker run --rm -it --memory=48g --entrypoint /bin/bash --volume ~/cuemin_results:/opt/results cuemin-experiments:latest -c 'bash /scripts/synthetic_fcfs_lcfs_with_batching.sh && bash /scripts/plot_results_of_synthetic_experiments_docker.sh'
```

```
docker run --rm -it --memory=48g --entrypoint /bin/bash --volume ~/cuemin_results:/opt/results cuemin-experiments:latest -c 'bash /scripts/synthetic_pq_no_batching.sh && bash /scripts/plot_results_of_synthetic_experiments_docker.sh'
```

## Running the Experiments on Real-World Datasets

The basic command to run the experiments on real-world data is

```
docker run --rm -it --memory=32g --entrypoint /bin/bash --volume ~/cuemin_results:/opt/prolothar-queue-mining/experiments/evaluation/results cuemin-experiments:latest -c 'python ${PYTHONPATH}/experiments/evaluation/run_on_single_queue_dataset.py --seed 42 --data_folder "experiments/datasets/Steel C" --train_until 96388 --categorical_features cat_0 --numerical_features num_0 num_1 num_2 --departure_time_unit s --sojourn_time_unit s --time_factor 1 --methods CueMin AW+DT AW+RF K-PHF RF RAS --nr_of_simulation_runs_for_prediction 1000
```

You will find the results under "~/cuemin_results". Feel free to play around with the parameters:

| Parameter                  | Description                                                       |
| -------------------------- | -----------------------------------------------------             |
| data_folder                | path to a real-world dataset. can be your own. see below for information on the required data format |
| train_until                | time step that separates train and test set                       |
| categorical_features       | space separated names of categorical features in the dataset      |
| numerical_features         | space separated names of numerical features in the dataset        |
| departure_time_unit        | defines the time unit of interest for the departure time. used in generated plots |
| sojourn_time_unit          | defines the time unit of interest for the sojourn time. used in generated plots |
| time_factor                | original time factor of the time steps in the dataset. 1 means seconds, 60 means minutes, 3600 means hours. this time factor is used in combination with departure_time_unit and sojourn_time_unit |
| seed                       | sets seeds for random number generators                           |
| experiment_name            | prefix of the result folder                                       |
| nr_of_simulation_runs_for_prediction | number of simulation runs to generate predictions (for CueMin and K-PHF) |
| methods                    | space separated list of methods for evaluation                    |

Run
```
docker run --rm -it --memory=32g --entrypoint /bin/bash --volume ~/cuemin_results:/opt/prolothar-queue-mining/experiments/evaluation/results cuemin-experiments:latest -c 'python ${PYTHONPATH}/experiments/evaluation/run_on_single_queue_dataset.py --help'
```
to see the complete list of available parameters.

We provide a script to run the same set of experiments we ran for our paper:

```
docker run --rm -it --memory=48g --entrypoint /bin/bash --volume ~/cuemin_results:/opt/results cuemin-experiments:latest -c 'bash /scripts/real_world_datasets.sh'
```

## Run CueMin on Your Data

Use the "data_folder" parameter in the real-world experiments to run CueMin on your data. The folder name is used as the name of your dataset. It should contain the following files:

### arrivals.txt

Contains a line for each arriving job (ordered by timestep):
```
integer timestep;job_id
```
We recommend to start with timestep 0. Job IDs must be unique!

### departures.txt

Contains a line for each exiting job (ordered by timestep):
```
integer timestep;job_id
```

### jobs.txt

Contains a line for each arriving or exiting job, e.g.:
```
Job(AA0101 43367, {'priority': '0', 'weekday': 'Sunday', 'hour': 7})
```
where "AA0101 43367" is the job id, which is followed by a comma and a set of
feature names with feature values.

### waiting_times.txt

This file is optional. It contains a line with the actual waiting time
for each job in the dataset.
```
job id;integer waiting time
```

## Questions?

Feel free to write us an email to ask any questions about our method, code, experiments, ...