Composite Block Types

Composite block types are a way to create new block types in Jayvee by combining the functionality of existing blocks and pipes. By relying on composite block types instead of implementing more built-in block types in a language interpreter, Jayvee supports easy extension by users.

Composite block types define:

• with the property keyword: properties with a name and value type, optionally a default value
• with the input keyword: one input with a name and io type (that can be None)
• with the output keyword: one output with a name and io type (that can be None)
• one pipeline definition, starting from the input (using its name) and ending in the output (again using its name)
• all blocks that are used in the pipeline definition (either blocks of built-in or composite block types)

Example

As an example, the common use-case of extracting a CSV file from a web server using HTTP. With built-in block types, a pipeline would start with a HttpExtractor source that downloads a file from the internet and outputs a binary file. This file must be interpreted as text (using a TextFileInterpreter) and finally as Sheet (using a CSVInterpreter).

Implementation with built-in block types

A pipeline with blocks using built-in block types is very verbose:

pipeline CarsPipeline {
    CarsExtractor
        -> CarsTextFileInterpreter
        -> CarsCSVInterpreter
        -> CarsTableInterpreter
        -> CarsLoader;

    block CarsExtractor oftype HttpExtractor {
        url: "https://example.com/cars.csv";
    }

    block CarsTextFileInterpreter oftype TextFileInterpreter { }

    block CarsCSVInterpreter oftype CSVInterpreter {
        enclosing: '"';
    }
    // ... further block definitions
}

Refactoring using composite block types

The common use-case of downloading a CSV file using HTTP can be refactored into a composite block type. Note that we define all properties of the built-in blocks that are used as properties of the new CSVExtractor block type (but add fallback values). If some internal configuration is always the same, we could also not expose it as a property of the new block type.

// Define a new composite block type named CSVExtractor outside of the pipeline
composite blocktype CSVExtractor {
    // Properties of the CSVExtractor, some with default values
    property url oftype text;
    property delimiter oftype text: ',';
    property enclosing oftype text: '';
    property enclosingEscape oftype text: '';

    // Input and outputs
    input inputName oftype None;
    output outputName oftype Sheet;

    // Pipeline definition from input, over blocks defined later, to output
    inputName
        ->FileExtractor
        ->FileTextInterpreter
        ->FileCSVInterpreter
        ->outputName;

    // Block definitions using values from properties by name
    block FileExtractor oftype HttpExtractor { url: url; }
    block FileTextInterpreter oftype TextFileInterpreter {}

    block FileCSVInterpreter oftype CSVInterpreter {
        delimiter: delimiter;
        enclosing: enclosing;
        enclosingEscape: enclosingEscape;
    }
}

With the new CSVExtractor composite block type, the pipeline now looks like this.

If the CSVExtractor is available in the scope of the CarsPipeline from before (e.g., by defining it above the pipeline), it can then be used to shorten the actual pipeline code.

pipeline CarsPipeline {
    // HttpExtractor, TextFileInterpreter and CSVInterpreter have been replaced by CSVExtractor
    CarsExtractor
        -> CarsTableInterpreter
        -> CarsLoader;

    block CarsExtractor oftype CSVExtractor {
        url: "https://example.com/cars.csv";
    }

    // ... further block definitions
}