Core Concepts

The core concepts of Jayvee are Pipelines, Blocks, and ValueTypes.

Pipelines

A Pipeline is a sequence of different computing steps, the Blocks. The default output of a block becomes the default input of the next block, building a chain of computing steps. In the scope of a Pipeline, you can connect these blocks via the pipe syntax:

pipeline CarsPipeline {
    // Assumption: blocks "GasReserveHttpExtractor", "GasReserveCSVInterpreter", "GasReserveTableInterpreter", and "GasReserveLoader" are defined

    GasReserveHttpExtractor
        -> GasReserveTextFileInterpreter
        -> GasReserveCSVInterpreter
        -> GasReserveTableInterpreter
        -> GasReserveLoader;
}

Blocks

A Block is a processing step within a Pipeline. It can have a default input and a default output. We differentiate the following types of Blocks:

• ExtractorBlocks do not have a default input but only a default output. They model a data source.
• TransformatorBlocks have a default input and a default output. They model a transformation.
• LoaderBlocks do have a default input but nor a default output. They model a data sink.

The general structure of a Pipeline consisting of different blocks is the following:

The common syntax of blocks is at its core a key-value map to provide configuration to the block. The availability of property keys and their respective ValueTypes is determined by the type of the Block - indicated by the identifier after the keyword oftype:

block GasReserveHttpExtractor oftype HttpExtractor {
    // key: value
    url: "https://www.bundesnetzagentur.de/_tools/SVG/js2/_functions/csv_export.html?view=renderCSV&id=1089590";
} 

In the example above, the url property of type text is defined by the corresponding HttpExtractor block type.

Blocks can be either defined as part of the language, called builtin or defined as composition of existing blocks by users in Jayvee, called composite. See the documentation for Composite Blocks.

ValueTypes

A ValueType is the definition of a data type of the processed data. Some Blocks use ValueTypes to define logic (like filtering or assessing the data type in a data sink). We differentiate the following types of ValueTypes:

• Built-in ValueTypes come with the basic version of Jayvee. See Built-in Valuetypes.
• Primitive ValueTypes can be defined by the user to model domain-specific data types and represent a single value. Constraints can be added to a Primitive ValueType. See Primitive Valuetypes.
• Compound ValueTypes: UPCOMING.

valuetype GasFillLevel oftype integer {
    constraints: [ GasFillLevelRange ];
}

constraint GasFillLevelRange on decimal:
    value >= 0 and value <= 100;

Transforms

Transforms are used to transform data from one ValueType to a different one. For more details, see Transforms.

transform CelsiusToKelvin {
  from tempCelsius oftype decimal;
  to tempKelvin oftype decimal;

  tempKelvin: tempCelsius + 273.15;
}