README

A Symfony bundle to build reliable, composable data pipelines (Read → Process → Write) with minimal boilerplate.

Features

• 📥 CSV/API/Custom Readers: pull data from files, HTTP APIs, DB, or any source
• 🔧 Processor Orchestration: rename indexes + validate rows before writing
• 📤 Writers out of the box: Doctrine ORM writer; easy to add file/API writers
• 🔁 Transformers: regular and postponed steps for clean normalization/hydration
• 🧩 Small, testable building blocks: plug only what you need

Installation

1. Require the bundle

composer require theovauvilliers/data-flow-bundle

2. Enable the bundle

Add the bundle to config/bundles.php:

return [
    // ... other bundles
    TheoVauvilliers\DataFlow\TheoVauvilliersDataFlowBundle::class => ['all' => true],
];

3. Configure Doctrine ORM

Create or update config/packages/doctrine.yaml:

doctrine:
   orm:
      entity_managers:
         default:
            mappings:
               # ... your existing mappings
               TheoVauvilliersDataFlowBundle:
                  type: attribute
                  is_bundle: false
                  dir: '%kernel.project_dir%/vendor/theovauvilliers/data-flow-bundle/src/Entity'
                  prefix: 'TheoVauvilliers\DataFlow\Entity'
                  alias: TVDataFlow

4. Configure Doctrine Migrations

doctrine_migrations:
    migrations_paths:
        # ... your existing paths
        'TheoVauvilliers\DataFlowBundle\Migrations': '%kernel.project_dir%/vendor/theovauvilliers/data-flow-bundle/migrations'

5. Configure routes

Create a file config/routes/theovauvilliers_data_flow.yaml and add the following content:

theovauvilliers_data_flow:
    resource: '@TheoVauvilliersDataFlowBundle/src/Controller/'
    type: attribute
    prefix: /{_locale}/data-flow

You can replace prefix by any other path you want. You can also remove the locale prefix if you don't use it.

6. Run database migration

Execute the bundle's migration to create the tracking table:

php bin/console doctrine:migrations:migrate

Usage

Create a Reader and a Writer, then run the Processor:

use TheoVauvilliers\DataFlow\Processor\Processor;
use App\DataFlow\Reader\PoiReader;
use App\DataFlow\Writer\PoiWriter;

$processor->process($reader, $writer, 'name of the import');

Typical flow: upload a CSV to a temp path your Reader expects, then call process().

How It Works

1. 📥 Reader returns an array of rows (CSV/API/DB/etc.).
2. 🧰 Processor loops rows, applies renameIndexes() and validate() from the Reader.
3. 📤 Writer persists/emits each row (Doctrine ORM, files, APIs…).
4. ✅ Processor calls Writer end() for finalization.
5. 🔄 Optional Transformers run pre/post hydration for clean domain data.

Architecture

• ReaderInterface + AbstractReader
  • AbstractCSVReader: BOM handling, optional header, delimiter auto-detection, empty-line skipping, normalization
  • AbstractApiReader: based on Symfony HTTP Client Contracts
• WriterInterface + AbstractWriter
  • AbstractDatabaseWriter: load-or-create, batch flushing, simple upsert via getFindOneParams()
• Processor: orchestrates Reader → (rename + validate) → Writer and calls end()
• Transformers: regular and postponed transformers for staged normalization

Best Practices

1. Keep Readers/Writers focused on a single dataset/entity
2. Use getIndexesToRenameOrUnset() to map input headers quickly
3. Add Symfony Validator constraints in Reader to skip invalid rows early
4. Compose small Transformers instead of big all-in-one mappers
5. For large imports, tune batch size (MAX_FLUSH) in AbstractDatabaseWriter

Run tools

PHPUnit

docker compose run --rm php vendor/bin/phpunit -c phpunit.xml.dist --colors=always

PHPStan

docker compose run --rm php vendor/bin/phpstan analyse -c phpstan.neon --no-progress --ansi --memory-limit=1G

php-cs-fixer

# check (no changes)
docker compose run --rm php vendor/bin/php-cs-fixer fix --config=.php-cs-fixer.php --dry-run --diff --ansi
# fix in place
docker compose run --rm php vendor/bin/php-cs-fixer fix --config=.php-cs-fixer.php --ansi

Composer

docker compose run --rm composer update

Tips

• Cache: add -v $(pwd)/.cache:/tmp/cache to persist tool caches between runs.
• If your service names differ, replace php/composer accordingly.