README

This library will generate jobs metadata obtained from Job Queue API open-api-lineage endpoint for a data lineage service supporting OpenLineage API (e.g. Marquez).

Installation

The library is available as composer package. To start using this library in your project follow these steps:

Install package:

composer require keboola/openlineage-generator

Add autoloader in your bootstrap script:

require 'vendor/autoload.php';

Read more in Composer documentation

Usage

$openLineageWriter = new OpenLineageWriter(
            $queueClient, // keboola job queue client
            $openLineageClient, // http client for open lineage api
            $logger,
            $createdTimeFrom, //DateTimeImmutablee object from which jobs will be imported
            $config->getOpenLineageEndpoint(), // url of open lineage api
            $config->getJobNameAsConfig() // boolean, if true, job name will be used as config name
        );

$openLineageWriter->write();

Development

Clone this repository and init the workspace with following command:

git clone https://github.com/keboola/openlineage-generator
cd openlineage-generator
docker-compose build
docker-compose run --rm dev composer install --no-scripts

Run the CI checks using this command:

docker-compose run --rm dev composer ci

Integration

For information about deployment and integration with KBC, please refer to the deployment section of developers documentation

License

MIT licensed, see LICENSE file.