test.js/README.md

# test.js

Combinational test framework.

This is not meant as a replacement for more advanced and feature-rich testing
frameworks, instead this is a minimalist and complete experimental implementation
of a specific approach to testing, i.e. _combinational testing_

Note, **this module is experimental** and can change quite allot within a short to mid
time frame, use at your own risk, though ideas, feedback and suggestions are welcome.


## Features

- Simple / minimalist implementation
- Supports combinational as well as unit testing paradigms
- Multiple modifier chaining
-

<!-- XXX file runner and it's logic... -->


### Planned

- Replaceable/extensible assertion library
-


## Contents
- [test.js](#testjs)
  - [Features](#features)
    - [Planned](#planned)
  - [Contents](#contents)
  - [Architecture](#architecture)
    - [Combinational testing](#combinational-testing)
    - [Unit testing](#unit-testing)
  - [Installation](#installation)
  - [Basic usage](#basic-usage)
  - [CLI](#cli)
  - [Components](#components)
    - [`DEFAULT_TEST_FILES`](#default_test_files)
    - [`IGNORE_TEST_FILES`](#ignore_test_files)
    - [`Merged`](#merged)
      - [`<merged>.create(..)`](#mergedcreate)
      - [`<merged>.members`](#mergedmembers)
      - [`<merged>.size` / `<merged>.usize`](#mergedsize--mergedusize)
      - [`<merged>.add(..)` / `<merged>.remove(..)`](#mergedadd--mergedremove)
      - [`<merged>.clear()`](#mergedclear)
      - [`<merged>.keys(..)` / `<merged>.values(..)` / `<merged>.entries(..)`](#mergedkeys--mergedvalues--mergedentries)
      - [`<merged>.toObject(..)`](#mergedtoobject)
      - [`<merged>.checkShadowing(..)`](#mergedcheckshadowing)
      - [`<merged>.handleShadowing(..)`](#mergedhandleshadowing)
      - [`<member>.filename`](#memberfilename)
    - [`TestSet`](#testset)
    - [`BASE_TEST_SET`](#base_test_set)
    - [`Setups(..)` / `Setup(..)` (Merged)](#setups--setup-merged)
    - [`Modifiers(..)` / `Modifier(..)` (Merged)](#modifiers--modifier-merged)
    - [`Tests(..)` / `Test(..)` (Merged)](#tests--test-merged)
    - [`Cases(..)` / `Case(..)` (Merged)](#cases--case-merged)
    - [`Assert(..)`](#assert)
    - [`run(..)`](#run)
  - [Advanced components](#advanced-components)
    - [`runner(..)`](#runner)
    - [`parser(..)`](#parser)
  - [Utilities](#utilities)
    - [`getCallerFilename()`](#getcallerfilename)
  - [License](#license)


## Architecture

This package implements two testing schemes:
- Combinational testing
  Here the user sets up a set of `Setup`, `Modifier` and `Test` functions and the system
  chains different combinations of the three and runs them.
- Unit testing
  Simple independent tests.


### Combinational testing

In general the idea here is that you define three things:

 - `Setups` that build a test context and objects (the _setup_),
 - `Modifiers` that modify the _setup_ in some way,
 - `Tests` that test some aspect of a _setup_.

The system builds chains in the form:
```
setup -> modifier* -> test
```

Where `modifier` is optional and can be a chain of zero or more modifiers.

A `setup` and `modifier` can also include assertions/tests for direct testing and
sanity checks.

The system defines a blank pass-through modifier and test, to alleviate the requirement
on the user to define at least one modifier and test, so in cases where it makes no
sense only a setup is required.

Note that each element can reference and re-use other elements so for example a
modifier can call (re-use) other modifiers to avoid code duplication.

This makes it simple to define procedural/generative tests.


### Unit testing

This is the traditional self-contained test approach.


## Installation

```shell_session
$ npm install -i ig-test
```

And to install the global CLI interface
```shell_session
$ npm install -g ig-test
```


## Basic usage

Create a test script
```shell_session
$ touch test.js
$ chmod +x test.js
```

Note that the test script should be named either `"test.js"` or `"<something>-test.js"`
for the system to find it automatically.

The code:
```javascript
#!/usr/bin/env node

var test = require('ig-test')

// XXX add assert examples...

test.Setups({
    state: function(assert){
        return {
            a: 123,
            b: 321,
        } },
})

test.Modifiers({
    inc: function(assert, state){
        Object.keys(state)
            .forEach(function(k){
                state[k] += 1 })
        return state },
})

test.Tests({

})

test.Cases({

})

// make the test runnable as a standalone script...
__filename == (require.main || {}).filename
    && tests.run()
```

Run the tests
```shell_session
$ node ./test.js
```

or
```shell_session
$ runtests
```

## CLI

```shell_session
$ npm install -g ig-test
```

Basic help
<!-- NOTE: to update this do :.,/'''/-1!./test.js --help -->
```shell_session
$ runtests --help
Usage: test.js [OPTIONS] [CHAIN] ...

Run tests.

Tests run by test.js can be specified in one of the
following formats:

        <case>
        <setup>:<test>
        <setup>:<modifier>:<test>

Each of the items in the test spec can be a "*" indicating
that all relevant items should be used, for example:

        $ ./test.js basic:*:*

Here test.js is instructed to run all tests and modifiers
only on the basic setup.

Zero or more sets of tests can be specified.

If no tests are specified or '**' is given test.js will run
all found tests:

        $ ./test.js
or:
        $ ./test.js '**'

Note that using '*' or '**' may require escaping as they may
get expanded by the shell.

Options:
	-h,  --help		- print this message and exit
	     --version		- show test.js version and exit
	-l,  --list=PATH	- list available tests;
				  note that if passing files via -f explicitly they
				  must precede the -l/-list flag;
				  this has the same defaults as -f
	     --list-found=PATH	- like -list but print found test modules and exit
	-m,  --max-modifier-chain=NUMBER
				- Maximum number of modifiers to use in chain
				  (default: 1)
	-f,  --test-file=PATH	- test script or filename pattern, supports glob;
				  this flag can be given multiple times for
				  multiple paths/patterns
				  (default: "**/?(*-)test.js")
	-i,  --ignore=PATH	- path/pattern to ignore in test file search
				  (default: ["node_modules/**"])
	-v,  --verbose		- verbose mode
				  (env: $VERBOSE)

Examples:
	$ ./test.js		- [90mrun all tests.[39m
	$ ./test.js basic:*:*	- [90mrun all tests and modifiers on "basic" setup.[39m
				  [90m(see test.js -l for more info)[39m
	$ ./test.js -v example	- [90mrun "example" test in verbose mode.[39m
	$ ./test.js native:gen3:methods init:gen3:methods
				- [90mrun two tests/patterns.[39m

Written by: Alex A. Naanou <alex.nanou@gmail.com>
Version: 1.6.7 / License: BSD-3-Clause
```

List available test components
```shell_session
$ runtests --list

```

XXX chains

XXX notes on coverage


## Components

### `DEFAULT_TEST_FILES`

[`glob`][glob] pattern(s) used to find test files by default.
```
DEFAULT_TEST_FILES =
    undefined
    | <path>
    | [ <path>, .. ]
```

Default value: `"**/?(*-)test.js"`


### `IGNORE_TEST_FILES`

A list of [`glob`][glob] patterns to ignore while searching for tests.
```
IGNORE_TEST_FILES =
    undefined
    | [ <path>, .. ]
```

Default value: `['node_modules/**']`


### `Merged`

Implements a _merged_ collection of instances (_members_).

Create a new collection:
```
Merged.create(<name>)
    -> <merged>
```

Add members to collection:
```
<merged>({ <key>: <func>, .. })
    -> <member>

<merged>(<key>, <func>)
    -> <member>
```

On construction this will assign the input object / `<key>`-`<func>` into the resulting
`<member>`/instance object.

Each `<member>`/instance created is added to the constructor as a _member_ (i.e.
added into `.members`)

Provides a set of methods and properties to access/introspect the _merged_
(hence the name) attributes of the _members_ (i.e. `.keys(..)`, `.values(..)`,
`.entries(..)`, `.size`/`.usize` and `.members`).

Note that though `Merged` itself is a collection, it is not designed to be used
directly as a collection, use it to create new collections or as a prototype for
inheritance.


#### `<merged>.create(..)`

Create a new `Merged` collection (member constructor).
```
Merged.create(<name>)
    -> <merged>
```


#### `<merged>.members`

List of _members_ / instances of `Merged` in order of creation.


#### `<merged>.size` / `<merged>.usize`

Number of _members_ including the `"-"` members and not including respectively.

#### `<merged>.add(..)` / `<merged>.remove(..)`

Add / remove a member.
```
<merged>.add(<member>)
    -> <merged>

<merged>.remove(<member>)
    -> <merged>
```

#### `<merged>.clear()`

Remove (_clear_) all the members.


#### `<merged>.keys(..)` / `<merged>.values(..)` / `<merged>.entries(..)`

```
<merged>.keys()
<merged>.keys(<merged>)
    -> <list>

<merged>.values()
<merged>.values(<merged>)
    -> <list>

<merged>.entries()
<merged>.entries(<merged>)
    -> <list>
```

These are similar to `Object.keys(..)` / `Object.values(..)` / `Object.entries(..)`
but will also if called without arguments return a list of the callers member
keys/values/entries respectively.

Note that members' attributes can _shadow_ previous member attributes, only one
value per key will be returned. `<merged>` will warn when adding a member of its
attributes will _shadow_ already existing members' attributes (see:
[`<merged>.checkShadowing(..)`](#mergedcheckshadowing) and
[`<merged>.handleShadowing(..)`](#mergedhandleshadowing));
Also note that the check for shadowing is performed when the `<member>` is
created and not when new attributes are added manually.


#### `<merged>.toObject(..)`

Create an object containing all visible member attributes.
```
<merged>.toObject()
    -> <object>
```

#### `<merged>.checkShadowing(..)`

Find all shadowed attributes within `<merged>`.
```
<merged>.checkShadowing()
```

Find all attributes in `<merged>` that will be shadowed by `<member>`
```
<merged>.checkShadowing(<member>)
    -> <list>
```


#### `<merged>.handleShadowing(..)`

Will be called on `<member>` construction when attribute _shadowing_ is detected.
```
`<merged>.handleShadowing(<attr>)`
    -> <merged>
```

By default this will print a warning and continue, but can be overloaded by the
user to react to _shadowing_ in a different manner.


#### `<member>.filename`

The filename where the `<member>` was defined.


### `TestSet`

XXX


### `BASE_TEST_SET`

XXX


### `Setups(..)` / `Setup(..)` (Merged)

XXX

A _subclass_ or rather _sub-constructor_ of `Merged`.

Note that `Setups` and `Setup` are references to the same object, they exists
for better readability in cases when we add a single element (`<key>`-`<func>`
pair) or a bunch of elements (object), for example:
```javascript
// single element...
test.Setup('some-setup',
    function(){
        // ...
    })

// arbitrary number of elements...
test.Setups({
    'some-other-setup': function(){
        // ...
    },

    // ...
})
```

### `Modifiers(..)` / `Modifier(..)` (Merged)

XXX

A _sub-constructor_ of `Merged`.

### `Tests(..)` / `Test(..)` (Merged)

XXX

A _sub-constructor_ of `Merged`.

### `Cases(..)` / `Case(..)` (Merged)

XXX

A _sub-constructor_ of `Merged`.


### `Assert(..)`

XXX this may still change...


### `run(..)`

Run the test system.
```
run()
run(<tests>)
run(<default-files>)
run(<default_files>, <tests>)
    -> <parse-result>
```

This will:
- parse `process.argv`
- locate and run tests
- report basic stats

`<tests>` format:
```
{
    setups: <stups>,
    modifiers: <modifiers>,
    tests: <tests>,
    cases: <cases>,
}
```


## Advanced components

### `runner(..)`

The default test combinator and runner.


### `parser(..)`

The default [`ig-argv`][ig-argv] parser setup.


## Utilities

### `getCallerFilename()`

Returns the filename of the module where `getCallerFilename()` is called.

- Multiple modifier chaining


## License

[BSD 3-Clause License](./LICENSE)

Copyright (c) 2020-2023, Alex A. Naanou,
All rights reserved.


<!-- External links -->
[glob]: https://github.com/isaacs/node-glob
[object.js]: https://github.com/flynx/object.js
[ig-argv]: https://github.com/flynx/argv.js


<!-- vim:set ts=4 sw=4 spell : -->