# SUSAN
# SUSAN
This is the accompanying distribution of the source code used for the purposes of this SDM submission.

## Installation

This source package is configured to use the standard *distutils* mechanism, which is able to build various distributables. Importantly, these include the *wheel* format. The latter can thus be used with pip, easy_install, conda and the like.

It is first advised to create a wheel file, which also allows a clean uninstall. To use this you may run
```
python setup.py bdist_wheel
pip install dist/cofi-0.5-*.whl
```
The source code is also configured to use the cmake system, which can be used in the `src` directory by issuing
```
mkdir -p build && cd build
cmake ../ -DCMAKE_BUILD_TYPE=Release -DPYTHON_EXECUTABLE=`which python`
make pycofi_impl
```
This mechanism is meant for development only. Therefore, the resulting binary python package `pycofi_impl` must be added to the PYTHONPATH manually.
### Requirements
Our code is known to work on linux debian systems with `g++ 8.3.0` and `cpython 3.6`. Further requirements can be installed through the `conda` and/or `pip` systems. Requirements include the following packages:
- numpy, scipy, pandas
- scikit-optimize, scikit-learn, grakel
- graph-tool
- recordclass

Optionally, the WWL package can also be provided, alongside its own dependencies.

To build the binary package you also need the development dependencies
- eigen (most recent/development version, providing begin/end iterators)
- pybind11

## Usage
The main functionality can be accessed through the Experiment class of the `cofi.experiments` module, which is responsible to interactively load
* a dataset (`load_dataset`)
* a kernel (`load_kernel`)
* an evaluator (`load_evaluator`)
* a cross validator (`load_xvalidator`)
but also to
* invoke the evaluation (`evaluate`)
* and optionally write the result (`summarise_to_json`)

A basic configuration appears in the following snippet:
```python
from skopt import spaces as ss
from cofi.experiments import Experiment
e = Experiment() 
e.load_dataset('morris:MUTAG') 
e.load_kernel('core-random-walk') 
e.load_estimator('ksvc',use_kernel=False) 
spaces = {'C':ss.Real(1e-4,1e5,'log-uniform'),'lamda':ss.Real(1e-4,1,prior='log-uniform')}
e.load_xvalidator('gp',spaces)
e.evaluate() 
js = e.summarise_to_json()
```
There are two commandline entry points for the package, the python scripts `susan-run.py` and `susan-results.py`. The former is a command-line exposure of the Experiment API, while the latter digests a list of result files into accessible CSV files. Both scripts provide extensive documentation of their commandline parameters.
There are two commandline entry points for the package, the python scripts `susan-run.py` and `susan-results.py`. The former is a command-line exposure of the Experiment API, while the latter digests a list of result files into accessible CSV files. Both scripts provide extensive documentation of their commandline parameters.
There are two commandline entry points for the package, the python scripts `susan-run.py` and `susan-results.py`. The former is a command-line exposure of the Experiment API, while the latter digests a list of result files into accessible CSV files. Both scripts provide extensive documentation of their commandline parameters.
There are two commandline entry points for the package, the python scripts `susan-run.py` and `susan-results.py`. The former is a command-line exposure of the Experiment API, while the latter digests a list of result files into accessible CSV files. Both scripts provide extensive documentation of their commandline parameters.

Both scripts contain multiple sample invocations on the body of their entry point, sample of which can be seen below.

```
# A simple invocation, corresponding to the code above
susan-run.py \
susan-run.py \
    load-dataset -p morris:MUTAG \
    load-kernel -n core-random-walk \
    load-estimator -n ksvc -U \
    load-xvalidator -n gp -R C:1e-1,5e4 -R lamda:1e-3,.99 -u \
    evaluate \
    summarise-json -o summary.json
```

```bash
# A more advanced invocation
susan-run.py \
susan-run.py \
	config -l PROGRESS `# Select progress debug level` \
    	-t MUTAG `#Select MUTAG as the "tag" of the experiment` \
        -D PROGRESS 10 `# Only print PROGRESS log messages every 10 seconds` \
        `#Set paths for each file type` \
        -p DATA data/grakel -p LOG output/logs/ -p WORK output/work -p CACHE output/cache -p RESULT output/results \
        load-dataset -p morris:Peking_1 `# Load the dataset of the morris database with name Peking_1` \
        load-kernel -n core-random-walk -D method_type=fast \
        	-Dkernel_type=geometric -Dtruncate=true \
            -Dvertex_kernel="incremental{'sidewidth':2}" \
            -C `# Enable caching of kernels, for use with the caching Grid XValidator` \
        load-estimator -n ksvc -Dmax_iter=1000 \
        	-U `# do not use the kernel here (but later on the xvalidator)` \
        load-xvalidator --with-train -n cg -K 10 -k 2 -s 2019 -R C:1e-1,5e4 -R lamda:1e-3,.99 -m  \
        	-u `# Make the XValidator kernel-aware, to not recompute the kernel` \
        evaluate -s 2019 --train-size 0.8 --test-size 0.2 -k 2 \
        summarise-json -GZ -o {config.tag_formatted}.json

```

There are 2 parts which can use the kernel: the estimator and the cross validator. Using it in one excluse use from the other. Usage from the xvalidator is the most advised, as the kernel is not re-computed for the parameters in the search spaces defined that do not affect it. However, this results in two loops of xvalidation iterations, an outer one that iterates over the parameters of the kernel, and an inner one that iterates over the rest (here, these should be the `C` parameters).

In addition, using the `cg` (caching grid-search) cross validator pre-computes the kernel on the entire dataset for each configuration.

In general, each experiment is repeated `k` times, where `k` is the `NUM_SPLITS` parameter of the `evaluate` action. For each of them, `NUM_CALLS` candidates are produced (configurable from the `load-xvalidator` action) for the kernel parameters, or for both the kernel and estimator, if the x-validator is kernel-agnostic. Otherwise, for each kernel parameter candidate, and extra loop of `NUM_CALLS_IN` is performed over candidates for the estimator parameters (typically its `C` parameter). For each candidate parameter the score is estimated using `k`-fold splitting, where `k` is specified as the `NUM_SPLITS` parameter of the `load-xvalidator` action.

Every experiment yields a *JSON* file which contains the final score, the description and parameters of all components (i.e., the evaluation, the kernel, xvalidator and estimator), various runtime information (e.g., hostname, username, pid, date, git version, etc) and also all statistics and results from the deeper iterations, including intermediate scores, kernel computation statistics and elapsed wall-clock times.

Although these results are well structured and can be parsed automatically, we found that it was convenient to agglomerate them in transactional tables in the form of `CSV` files, where each inner iteration is associated with its upper one with a relevant index, that functions equivalently to a private key in a relational database. This later operation is performed with the `susan-run.py` script. A sample invocation of it is as follows:
Although these results are well structured and can be parsed automatically, we found that it was convenient to agglomerate them in transactional tables in the form of `CSV` files, where each inner iteration is associated with its upper one with a relevant index, that functions equivalently to a private key in a relational database. This later operation is performed with the `susan-run.py` script. A sample invocation of it is as follows:
```bash
susan-results.py -P `# show progress` \
susan-results.py -P `# show progress` \
	-o digests/ `#output folder for the digests` \
    -j 10 `#number of parallel jobs` \
    -x digest- `#prefix for the generated tables` \
    results/*.json `#list of files to consider in the digest`
```


## Disclaimer

Although all our code is portable `python3` and `C++-17`, and should run in all systems implementing these languages, due to limited resources we can only guarantee its operation on `linux` systems. It is our observation that most scientific supercomputers are equipped with such an operating system and fulfill those basic requirements, and we therefore expect easy access to our code in the scientific community. What is more, all of these technologies are readily available free of cost (cpython, gcc-7), so we expect a reasonably easy use at minimal cost either directly or at least through basic virtualisation solutions, whose maintenance unfortunately exceed our capacity and scope.



