Pipeline

Enter the repository

Clone and enter the project repository.

git clone https://github.com/objectionary/normalizer --recurse-submodules
cd normalizer

Install normalizer

Install normalizer.

export LC_ALL=C.UTF-8
stack install eo-phi-normalizer

Install dependencies

Install yq.

yq --version

Install NodeJS.

node --version

Install Java.

java --version

Windows

Install Cygwin.

Make cygpath available on PATH.

Learn about the pipeline

Pipeline stages

The pipeline has several important stages:

  • Transform original EO programs into filtered EO programs, leaving only the specified top-level objects (tests).
  • Translate filtered EO programs to initial PHI programs.
  • Translate initial PHI programs to initial EO programs.
  • Test initial EO programs.
  • Normalize initial PHI programs and get normalized PHI programs.
  • Report metrics on initial PHI programs and normalized PHI programs.
  • Translate normalized PHI programs to normalized EO programs.
  • Test normalized EO programs.

Pipeline configuration

The pipeline is configured via the pipeline/config.yaml file.

The configuration file specifies the following:

  • report - Pipeline report configuration.
    • js - Optional path to a JavaScript file that should be inlined into the HTML report.
    • css - Optional path to a CSS file that should be inlined into the HTML report.
    • output - Where to write report versions.
      • html - The file path of the HTML version.
      • json - The file path of the JSON version.
      • markdown - The file path of the GitHub Flavored Markdown version.
    • expected-metrics-change - The expected relative change in metrics w.r.t the initial metrics.
      • dataless - For dataless formations.
      • applications - For applications.
      • formations - For formations.
      • dispatches - For dispatches.
    • expected-improved-programs-percentage - Expected percentage of programs where all metrics changed as expected.
  • test-sets - A list of configurations for sets of test objects (tests).
    • eo - The configuration of an EO test set.
      • original - The file path of the original EO program.
      • enable - A flag to enable tests in the original EO program.
      • include - A list of names of tests in the original EO program that should be included into the filtered EO program.
      • exclude - A list of names of tests in the original EO program that shouldn't be included into the filtered EO program.
      • filtered - The file path of the filtered original EO program.
      • yaml - The file path of the original EO program in the YAML format.
    • phi
      • initial: The file path of the initial PHI program.
      • normalized: The file path of the normalized PHI program.
      • bindings-path-initial: The path to tests via bindings in the initial PHI program.
      • bindings-path-normalized: The path to tests via bindings in the normalized PHI program.

Run the pipeline script

bash ./scripts/pipeline.sh

Explore the pipeline directory

The script will run for several minutes and write the following entries to the pipeline directory:

  • eo-filtered - Filtered EO programs.
  • eo-initial - Initial EO programs.
    • .eoc/4-pull/org/eolang - org.eolang objects (link).
  • eo-normalized - Normalized EO programs.
  • eo-yaml - Filtered EO programs in the YAML format.
  • logs - Logs of some pipeline stages.
  • phi-initial - Initial PHI programs.
    • .eoc/phi/org/eolang - org.eolang objects translated to PHI.
  • phi-normalized - Normalized PHI programs.
  • report - Pipeline reports.
    • report.html - The report in the HTML format.
    • report.json - The report in the JSON format.
    • report.md - The report in the GitHub Flavored Markdown format.