README

Introduction

This is the Symfony bundle part of the VDM.library used as the foundation to build VDM.collect, VDM.compute and VDM.store components.

It provides services to integrate your instances of VDM.collect, VDM.compute and VDM.store components with all other core component of the VDM platform (VDM.backbone, VDM.control and VDM.flow)

Services

• VDM.collect HTTP pull
• VDM.collect FTP/SFTP
• VDM.backbone consumer
• VDM.backbone producer
• VDM.store Elasticsearch
• VDM.store Doctrine

Features

• Collect HTTP sources :
  • Retry on error
  • Monitoring of HTTP Responses
• Produce message to a broker with exit on error
• Consume message from a broker with exit on error
• Handle Message between consumption and production
• Monitoring :
  • Track usage (number of messages consumed, produced)
  • Track state (started, stopped, errored)
  • Track process metrics (memory, processing time)
  • Track dependencies metrics
    • http : sources states, response codes, response times, response size
    • ftp : source states, errors and file size

These features are built on top of the Symfony Framework and its messenger component

Documentation

• Concepts
• Getting started
• Consume :
  • RabbitMQ json
  • HTTP pull
  • Kafka
  • FTP/SFTP
• Produce :
  • Kafka
  • Elasticsearch
  • Doctrine ORM
  • Doctrine ODM
• Manual Tranport
• Monitoring
• Configuration Reference
• Docker
• Examples :
  • Setup a VDM.collect instance to pull meteo data
  • Setup a VDM.compute instance to process meteo data
  • Setup a VDM.store instance to store meteo data

Image:

export DOCKER_VDM_LIBRARY_VERSION=3.0.1
docker build -t 3slab/vdm-library-base:$DOCKER_VDM_LIBRARY_VERSION .
docker push 3slab/vdm-library-base:$DOCKER_VDM_LIBRARY_VERSION

License

This bundle is distributed around the MIT License