1'''
  2A module for running experiments that involve the execution of
  3commands with different combinations of command line arguments.
  4
  5Read on for an explanation with a bunch of examples, check out the
  6[reference documentation](./run.html), or download the
  7[code](https://github.com/thobl/run).
  8
  9# Basic Concepts
 10
 111. **Generate Argument Combinations.** Instead of specifying all
 12   combinations of command line arguments for your experiment
 13   explicitly, you specify the sets of values for each argument.  A
 14   set of runs with one run for each combination is generated
 15   automatically.
 16
 171. **Everything is a Blob.** At its core a blob is something that
 18   evaluates to a string based on the arguments of a single run.  In
 19   most cases, a blob is simply a string that can contain wildcards
 20   for these arguments.
 21
 221. **Skip Unnecessary Runs.** Runs are skipped if the produced output
 23   file already exists.  This is particularly helpful if the
 24   experiment setup evolves over time.
 25
 261. **Postprocessing the Output.** If the command prints its result to
 27   ``stdout``, you can specify how it should be saved to a file.  This
 28   lets you, e.g., add some reformatting procedure or handle timeouts.
 29
 30# Usage
 31
 32Your experiment script should look somewhat like this:
 33
 34```python
 35# example.py
 36import run
 37
 38run.add(...)
 39run.add(...)
 40...
 41run.run()
 42```
 43
 44The method ``run.add()`` has three mandatory parameters and multiple
 45keyword parameters.  The first parameter specifies the name of the
 46experiment.  This is used so you can select which experiments to run,
 47e.g., calling ``python example.py name1 name2`` runs all experiments
 48where the name was ``name1`` or ``name2`` (experiments can also be
 49[grouped](#experiment-names-and-groups)).  The second parameter
 50specifies the command that should be called and the third parameter is
 51a description of the arguments the command should be called with.
 52Their usage is explained in the [next
 53section](#generate-argument-combinations).
 54
 55## Generate Argument Combinations
 56
 57Here is an easy example for an experiment that create some files by
 58calling the ``touch`` command.
 59
 60```python
 61run.add(
 62    "create_files",
 63    "touch output/a=[[a]]_b=[[b]]_c=[[c]].txt",
 64    {'a': [1, 2], 'b': [1, 2, 3], 'c': 0}
 65)
 66```
 67
 68The third parameter specifies the three arguments ``a``, ``b``, and
 69``c``.  As ``a`` and ``b`` are lists, one run is created for each
 70combination of arguments.  The second parameter specifies the command
 71for each run.  It is a blob, i.e., the wildcard of the form ``[[a]]``
 72is replaced with the corresponding value for ``a`` in each of the
 73runs.  Thus, the above experiment calls ``touch`` six times creating
 74the following files.
 75
 76```shell
 77$ ls output/
 78'a=1_b=1_c=0.txt'  'a=1_b=2_c=0.txt'  'a=1_b=3_c=0.txt'
 79'a=2_b=1_c=0.txt'  'a=2_b=2_c=0.txt'  'a=2_b=3_c=0.txt'
 80```
 81
 82**Note:** In case your experiment involves some dependencies between
 83the parameters that make some of the combinations invalid, you can
 84[filter those out](#filtering-the-combinations).
 85
 86## Everything is a Blob
 87
 88In the above example, the command (second parameter) is a blob, i.e.,
 89all wildcards are replaced with the appropriate values for each run.
 90Besides being a string with wildcards, a blob can also be a function
 91that returns a string with wildcards.  In this case, it is called with
 92the arguments of the current run as parameter, before the wildcards
 93are replaced.  See [Blobs](#blobs) for more details.
 94
 95Almost all parameters of `run.add()` are blobs and are thus evaluated
 96based on the arguments of each individual run.  Even the arguments
 97themselves can be blobs based on earlier arguments.  The [next
 98section](#skip-unnecessary-runs) contains an example for this.
 99Moreover, there are some more details on blobs [below](#blobs).
100
101## Skip Unnecessary Runs
102
103In the above example, ``run`` does not know that the runs create
104files.  Thus, the ``touch`` commands is called even if the files
105already exist.  You can use the optimal parameter ``creates_file`` to
106explicitly state which file is created by a run.  It is a blob, so in
107the above example you could add
108``creats_file="output/a=[[a]]_b=[[b]]_c=[[c]].txt"``.  Equivalently,
109you can achieve the same in the following more elegant way, using that
110the arguments themselves are blobs.
111
112```python
113run.add(
114    "create_files",
115    "touch [[file]]",
116    {'a': [1, 2], 'b': [1, 2, 3], 'c': 0,
117     'file': "output/a=[[a]]_b=[[b]]_c=[[c]].txt"},
118    creates_file="[[file]]"
119)
120```
121
122Now executing the script calls ``touch`` only for those files that do
123not already exist.  For more details on when exactly a run is skipped,
124see [Policy of Skipping Runs](#policy-of-skipping-runs).
125
126**Note:** Specifying ``creates_file`` is only necessary if the command
127itself creates the file.  If instead the command's output to ``stdout``
128is written to a file (see [next section](#postprocessing-the-output)),
129``run`` already knows which file is created.
130
131## Postprocessing the Output
132
133If the parameter ``stdout_file`` is provided, the command's output to
134``stdout`` is written to the corresponding file.  This is illustrated
135by the following example.
136
137```python
138run.add(
139    "sum",
140    "echo $(([[a]] + [[b]]))",
141    {'a': [1, 2], 'b': [1, 2, 3]},
142    stdout_file="output/[[a]]+[[b]].txt"
143)
144```
145
146The ``echo`` command itself adds the two given values and writes them
147to standard output.  The run module takes this result and writes it
148to the provided file instead.  Thus, executing this example creates
149six files with different values for a and b, each of which contains
150the sum of a and b as content.
151
152You can postprocess ``stdout`` (given as string) by providing the
153parameters ``stdout_mod`` (a function returning a blob) or
154``stdout_res`` (a blob).  If both are provided, they are applied in
155this order.
156
157The function ``stdout_mod`` gets ``stdout`` as input its return value
158replaces ``stdout``.  This can serve as some kind of parser that,
159e.g., reformats the output from something human readable to csv.
160Additionally, if the function takes two parameters, the second
161parameter is the result of the call to
162[``subprocess.run()``](https://docs.python.org/3/library/subprocess.html#subprocess.run).
163This can be useful to do a more sophisticated reformatting not only
164based on ``stdout`` but also, e.g., on the return code or stderr.
165
166If the blob ``stdout_res`` is given, this blob is written to the file
167instead of ``stdout`` itself.  This blob is somewhat special as it
168allows for the additional argument ``stdout``.  The following example
169uses ``stdout_res`` with the wildcard ``[[stdout]]``.
170
171```python
172run.add(
173    "sum",
174    "echo $(([[a]] + [[b]]))",
175    {'a': [1, 2], 'b': [3, 4]},
176    stdout_file="output/sums.txt",
177    stdout_res="[[a]] + [[b]] = [[stdout]]"
178)
179```
180
181This produces the following file.
182
183``` shell
184$ cat output/sums.txt 
1852 + 3 = 5
1862 + 4 = 6
1871 + 3 = 4
1881 + 4 = 5
189```
190
191Note that the output of the individual runs was appended to the file,
192i.e., the runs were not skipped.  This is the intended behavior to
193support these kind of experiment setups; see [Policy of Skipping
194Runs](#policy-of-skipping-runs) for more details on when exactly a run
195is skipped.
196
197**Note:** There are several cases where ``stdout_mod`` and
198``stdout_res`` can both be used to achieve the same thing.  You can
199use this rule of thumb:
200
201 * Use ``stdout_mod`` if you want to reformat ``stdout`` or if you
202   need access to things like the return code.
203
204 * Use ``stdout_res`` if you want to combine ``stdout`` with the input
205   arguments.
206
207 * Use both if you need both.
208 
209**Note:** When using ``stdout_file``, one can additionally [write a
210header](#headers) to the first line of the file.
211
212# Details
213
214When you execute your experiment script, the following steps are
215performed in this order.
216
2171. **Generate Combinations.** For each experiment, create the cross
218   product of all argument lists provided in third parameter of
219   ``run.add()``.  For each argument combination that is not rejected
220   by [``combinations_filter``](#filtering-the-combinations), a single
221   run is created.  Each individual argument is a blob.
222
2231. **Deblob.** For each individual run, the [blobs are turned into
224   strings](#blobs).  This is first done for all arguments in the
225   order they are given.  Then all other blobs (except ``stdout_res``)
226   are deblobbed.
227
2281. **Skipping Runs.** For each run, it is decided whether to skip the
229   run.  A run is skipped if the file it would produce (specified by
230   ``creates_file`` or ``stdout_file``) already exists at this stage
231   in the process.
232
2331. **Filter Unselected Runs.** A run counts as selected if its
234   experiments [name or group](#experiment-names-and-groups) was used
235   as command line parameter to execute your experiment script.  All
236   other runs are ignored.
237   
2381. **Run and Output Results.** The runs are executed and, depending on
239   the [stdout settings](#postprocessing-the-output) and the [header
240   settings](#headers), the output is written.
241
242
243## Blobs
244
245Deblobbing happens in three steps in the function ``run.deblob()``.
246
2471. If the blob is a function, it is called with one parameter
248   ``args``.  ``args`` is a dictionary mapping from the argument names
249   to the value.  Note that this is done after generating the
250   combinations, i.e., the argument values are no longer lists.
251
2521. The result of step 1. (or the blob itself if it is not a function)
253   is converted into a string.  Thus, it is ok for an argument to
254   have, e.g., integer values or for the above function to return
255   integer values.
256
2571. In the resulting string, each pattern of the form ``[[key]]`` is
258   replaced with the corresponding value in ``args``.
259
260The following example is similar to one of the previous examples,
261extending the computation of the sum to also compute differences,
262products, and quotients.  It gives an example of how to make use of
263the fact that blobs can be functions.  Note how the output file name
264(``stdout_file``) of each run depends on the operator.  As the
265argument combinations are generated before deblobbing,
266``args['operator']`` in the lambda function evaluates to one of the
267operators (and not to the list of all operators).  Running this,
268creates the four files ``sum.txt``, ``diff.txt``, ``prod.txt``, and
269``quot.txt``, each containing the respective calculations.
270
271```python
272res_name = {'+': "sum", '-': "diff", '*': "prod", '/': "quot"}
273run.add(
274    "calculate",
275    "echo $(([[a]] [[operator]] [[b]]))",
276    {'a': [1, 2], 'b': [3, 4],
277     'operator': ["+", "-", "*", "/"]
278     },
279    stdout_file=lambda args: "output/" + res_name[args['operator']] + ".txt",
280    stdout_res="[[a]] [[operator]] [[b]] = [[stdout]]"
281)
282```
283
284The next example highlights how the order of evaluation matters when
285using blobs for the arguments.  Assume we have three arguments ``a``,
286``b``, and ``c`` that depend on each other, i.e., we want to consider
287not all but only certain combinations of values.  In this example, we
288want ``a``, ``b``, and ``c`` to be all combinations of natural numbers
289with ``a + b + c = 4``.  In the example below, this is done using the
290helper argument ``triple``, which is a list of triples.  Then the
291actual arguments ``a``, ``b``, and ``c`` use these triples.  Note that
292blobs of ``a``, ``b``, and ``c`` are deblobbed (i.e., their lambda
293functions are called) after lists have been split into separate runs
294but before the ``triple`` argument was deblobbed.  Thus,
295``args['triple']`` is a dictionary of the form ``{'a': a, 'b': b, 'c':
296c}``.  Moreover, ``a``, ``b``, and ``c`` are deblobbed before
297``file``, which makes them usable as wildcards.
298
299```python
300run.add(
301    "sum_of_squares",
302    "echo $(([[a]] * [[a]] + [[b]] * [[b]] + [[c]] * [[c]]))",
303    {'a': lambda args: args['triple']['a'],
304     'b': lambda args: args['triple']['b'],
305     'c': lambda args: args['triple']['c'],
306     'triple': [{'a': a, 'b': b, 'c': c}
307                for a in range(0, 5) for b in range(0, 5) for c in range(0, 5)
308                if a + b + c == 4],
309     'file': "output/sum_of_squares_[[a]]_[[b]]_[[c]].txt"
310     },
311    stdout_file="[[file]]"
312)
313```
314
315Changing the order could have unwanted effects.  Moving the argument
316``triple`` before ``a`` would first deblob ``triple``, before calling
317the lambda function, which means that ``args['triple']`` would be the
318string representation of the dictionary instead of the dictionary
319itself.  Moreover, having ``file`` before ``a`` would make the use of
320``'[[a]]'`` invalid, as the argument ``a`` would not be a string but a
321function when trying to do the replacement.
322
323As a rule of thumb: Arguments you use as wild cards should go before,
324arguments you want to use in their original form (not in their string
325representation) within a function should go after.
326
327**Note:** In the above example one could of course remove the ``file``
328argument and directly assign its blob to ``stdout_file``, as the
329arguments are always deblobbed before any other blobs (like
330``stdout_file``).
331
332## Filtering the Combinations
333
334The previous example showed a somewhat hacky way of only looking at
335those combinations of arguments ``a``, ``b``, and ``c`` where ``a +
336b + c = 4``.  This can be achieved much easier using the parameter
337``combinations_filter``.  It is a function that gets called for every
338argument combination and decides whether it is valid (in which case a
339run is created) or not (in which case the combination is discarded).
340
341```python
342run.add(
343    "better_sum_of_squares",
344    "echo $(([[a]] * [[a]] + [[b]] * [[b]] + [[c]] * [[c]]))",
345    {'a': list(range(0, 5)), 'b': list(range(0, 5)), 'c': list(range(0, 5))},
346    stdout_file="output/sum_of_squares_[[a]]_[[b]]_[[c]]_good.txt",
347    combinations_filter=lambda args: args['a'] + args['b'] + args['c'] == 4
348)
349```
350
351## Policy of Skipping Runs
352
353From the [order of steps](#details) discussed above, it becomes
354implicitly clear that runs are only skipped if the result files
355already exist at the time the script is executed.  Thus, if two runs
356have the same non-existent output file, they are both run, even if the
357file exists when the second run starts.  To make this more explicit,
358here is the behavior of writing files when using the ``stdout_file``
359parameter.
360
3611. If the file exists when adding the experiment via ``run.add()``
362   (i.e., before running any experiment), then the run is skipped.
363
3641. If the run is not skipped but the file exists when executing the
365   run, ``stdout`` is added at the end of the file.
366
3671. If the file does not exist, it is created beginning with the
368   [header](#headers) (if specified) and then ``stdout`` is added.
369   
370Thus, instead of writing one file per run, one can also have multiple
371runs write their output into a single file.  This can be the intended
372behavior if, e.g., the output of the same algorithm on multiple
373instances should appear in the same csv file.  The disadvantage of
374this is that either all or none of the experiments creating this file
375are skipped, so extending the experiment would require to rerun it
376(which might be ok, depending on your use case).
377			   
378## Headers
379
380When using ``stdout_file``, one can use the parameter
381``header_string`` (a blob) to specify a header that should appear in
382the first line of the file.  Alternatively, one can specify
383``header_command`` (again, a blob), which is a command that is
384executed and its output (to ``stdout``) is used as header.  The
385resulting header can be modified by specifying the parameter
386``header_mod``.  It should be a function that takes the unmodified
387header as input and outputs the actual (modified) header.
388
389Using ``header_command`` instead of simply ``header_string`` is
390advantageous in the following scenario.  Assume you have an executable
391called ``algo`` that runs a certain algorithm on an input file and
392outputs some statistics using comma-separated values.  Moreover,
393assume ``algo`` is capable of printing the corresponding header by
394calling ``algo --only-header``.  Then the following example first
395writes the header (obtained by calling ``algo --only-header``)
396followed by the results of running ``algo`` on three input files.
397
398```python
399run.add(
400    "algo",
401    "algo [[input]]",
402    {'input': ["file1", "file2", "file3"]},
403    stdout_file="output.csv",
404    header_command="algo --only-header"
405)
406```
407
408The advantage of this is that the knowledge about the output of
409``algo`` is concentrated at the ``algo`` executable, i.e., the
410experiment script does not have to change if the statistics printed by
411``algo`` change.  Moreover, using ``header_command`` over
412``header_string`` becomes even more advantageous, when running
413multiple commands, e.g., ``algo1`` and ``algo2``, with the same
414command line interface (but potentially very different outputs) on the
415same set of files.  This is illustrated by the following example (note
416the usage of the wildcard ``[[algo]]`` in the command (second
417parameter) and the ``header_command``).
418
419```python
420run.add(
421    "algos",
422    "[[algo]] [[input]]",
423    {'algo': ["algo1", "algo2"],
424     'input': ["file1", "file2", "file3"]},
425    stdout_file="output_[[algo]].csv",
426    header_command="[[algo]] --only-header"
427)
428```
429
430## Error Handling
431
432This section is only about errors that happen when executing a run.
433If you have an error in your experiment specification, then you
434probably just get a python error you have to deal with.
435
436You can specify the allowed return codes of an experiment via the
437parameter ``allowed_return_codes``.  The default is ``[0]``.  If the
438return code of the command is not in the list of allowed return codes,
439a warning is printed and the run is aborted, meaning that ``stdout``
440is **not** written to the file given by ``stdout_file``.  You can set
441``allowed_return_codes`` to ``[]`` to indicate that every return code
442should be accepted (you probably should not do this).
443
444In case different return codes are possible, you probably want to
445postprocess the result by using ``stdout_mod`` or ``stdout_res``.
446With the former, you have access to the return code; with the latter
447you only have the output of the command; see the section about
448[postprocessing the standard output](#postprocessing-the-output).
449
450In the following example, ``sleep [[time]] && echo waking up`` waits
451for 0 to 4 seconds and then outputs ``waking up``.  The command
452``timeout 2`` aborts this after 2 seconds and returns with code
453``124`` and no output to ``stdout``.  The return codes ``0`` and
454``124`` are allowed by setting ``allowed_return_codes=[0, 124]``.
455Moreover, the blob ``stdout_res`` specifies the output depending on
456whether there was a timeout.
457
458```python
459run.add(
460    "timeouts",
461    "timeout 2 sleep [[time]] && echo waking up",
462    {'time': [0, 1, 2, 3, 4]},
463    stdout_file="output/timeouts.txt",
464    allowed_return_codes=[0, 124],
465    stdout_res=lambda args: (
466        "sleeping [[time]]s -> [[stdout]]" if args['stdout'] != "" else
467        "sleeping [[time]]s -> timeout")
468)
469```
470This produces the following file.
471
472```shell
473$ cat output/timeouts.txt 
474sleeping 0s -> waking up
475sleeping 1s -> waking up
476sleeping 3s -> timeout
477sleeping 4s -> timeout
478sleeping 2s -> timeout
479```
480
481The above example relies on the fact that the standard output is the
482empty string in case of an error.  The following yields the same
483result but explicitly checks for the return code using ``stdout_mod``
484instead of ``stdout_res``.
485
486```python
487run.add(
488    "timeouts",
489    "timeout 2 sleep [[time]] && echo waking up",
490    {"time": [0, 1, 2, 3, 4]},
491    stdout_file="output/timeouts.txt",
492    allowed_return_codes=[0, 124],
493    stdout_mod=lambda out, res: (
494        "sleeping [[time]]s -> [[stdout]]"
495        if res.returncode == 0
496        else "sleeping [[time]]s -> timeout"
497    ),
498)
499```
500
501In the above example, one could also use the parameter ``out``
502directly instead of using the wildcard ``[[stdout]]``.
503
504# Miscellaneous Features
505
506There are some features that are not at the core of `run` in the sense
507that they are independent of how you specify your experiments.
508Nonetheless they are pretty useful and should be mentioned here.
509
510## Experiment Names and Groups
511
512When running the experiment script with all examples from above by
513calling ``python example.py create_files calculate``, the output looks
514somewhat like this:
515
516![Example Run 1](example_run_1.png)
517
518You can see that a bunch of experiments are available.  The two
519selected ones are highlighted in green (here ``create_files `` and
520``calculate``), all others are red.
521
522To select which experiments to run more conveniently, you can group
523experiments using the ``run.group()`` function as follows.
524
525```python
526# example.py
527import run
528
529run.group("group_name_a")
530run.add("exp1", ...)
531run.add("exp2", ...)
532...
533run.group("group_name_b")
534run.add("exp3", ...)
535run.add("exp4", ...)
536...
537run.run()
538```
539
540By running the script with a group name as parameter, all experiments
541in that group are run, e.g., ``python example.py group_name_a`` runs
542the experiments "exp1" and "exp2".
543
544Experiment names are of course also blobs, so you can do the following
545to create experiments with four different names in one group by
546calling ``run.add()`` just once.
547
548```python
549res_name = {'+': "sum", '-': "diff", '*': "prod", '/': "quot"}
550run.group("calculations")
551run.add(
552    "calculate_[[op_name]]",
553    "echo $(([[a]] [[operator]] [[b]]))",
554    {'a': [1, 2], 'b': [3, 4],
555     'operator': ["+", "-", "*", "/"],
556     'op_name': lambda args: res_name[args['operator']]
557     },
558    stdout_file="output/result_[[op_name]].txt",1
559    stdout_res="[[a]] [[operator]] [[b]] = [[stdout]]"
560)
561```
562
563Adding some more groups and calling ``python example.py sum
564sum_of_squares calculations`` yields the following output.
565
566![Example Run 1](example_run_2.png)
567
568## Wildcard matching for experiment names
569
570When specifying the names of experiments to be run, it is also possible to use
571shell wildcards (* and ?).  Note that in most shells the names need to be put
572into quotes in order to not be evaluated before being passed to python.
573
574![Example Wildcard-Run 1](example_run_wildcards.png)
575
576## Parallelization
577
578Runs from the same experiment are run in parallel and you can choose
579how many threads to use by calling ``run.use_cores(nr_cores)``.  By
580default, ``4`` cores are used.
581
582**Note:** There is no parallelization between different experiments,
583i.e., in the above example, all runs from ``sum`` have to finish
584before the runs of ``sum_of_squares`` start.
585
586**Note:** Files are locked for writing, i.e., it is ok for multiple
587runs of the same experiment to write to the same file.  It is also
588made sure that the header is written only once.  If you abort an
589experiment by interrupting it with ``ctrl + c`` there might be a
590leftover ``*.lock`` file, which you have to remove manually.
591
592## Return String
593
594The method ``run.add()`` can return a list of strings, one for each
595run.  This can, e.g., be helpful if each run of the experiment
596generates a file and you want to perform an additional task on each of
597these files in a later experiment.
598
599To use this feature, set the parameter ``return_string`` of
600``run.add()``.  It is a blob.
601
602## Dry Run
603
604If you execute your experiment script with ``dry_run`` as parameter,
605the runs are not executed.  Instead it is printed to ``stdout`` which
606runs would have been called.
607
608## Creating Directories
609
610If you use ``stdout_file`` and the directory where the output file
611should be stored does not exist, it is created.
612
613## Multiple Runs in One Script
614
615You can call ``run.run()`` multiple times in one script.  Each call
616after the first considers only the runs added after the previous call.
617This is useful when there are some experiments that create files and
618then later experiments do something for each file created earlier.
619This is demonstrated in the following example, where ``create_files``
620creates some ``.txt`` files while ``copy_files`` copies all ``.txt``
621files that are found by
622[``glob.glob("output/*.txt")``](https://docs.python.org/3/library/glob.html#glob.glob).
623
624```python
625run.add(
626    "create_files",
627    "touch [[file]]",
628    {"a": [1, 2], "b": [1, 2, 3], "c": 0, "file": "output/a=[[a]]_b=[[b]]_c=[[c]].txt"},
629    creates_file="[[file]]",
630)
631run.run()
632
633import glob
634files = glob.glob("output/*.txt")
635
636run.add(
637    "copy_files",
638    "cp [[file]] [[copy]]",
639    {"file": files, "copy": "[[file]].copy"},
640    creates_file="[[copy]]",
641)
642run.run()
643```
644
645Without the first call to ``run.run()``,
646[``glob.glob()``](https://docs.python.org/3/library/glob.html#glob.glob)
647would not find the files created by ``create_file`` as they are not
648yet created.  Thus, one would have to call the script twice to do both
649experiments.  With the additional call to ``run.run()``, one can run
650``create_files`` and ``copy_files`` in one execution of the script.
651
652## Section Headlines
653
654You can call ``run.section()`` to print a section title (e.g., to
655structure the output in case of multiple run calls).
656
657# [Interface Documentation](./run.html)
658
659'''