actionETL

.NET ETL library combining the best of the ETL mindset with the tools & techniques of modern application development

News

actionETL is a high performance, highly productive .NET library for easily writing ETL data processing applications in .NET languages such as C# and VB.

Use it to add ETL processing to your existing applications, and to create new ones, replacing or augmenting traditional ETL tools.

It is suitable for ETL developers with limited as well as extensive .NET programming experience, and for full-time .NET developers that have ETL requirements.

The Effective Way to ETL, using .NET

Outstanding Features

High Performance

  • Combine in-memory row-by-row processing, database, and file-based processing to get the best of each approach
  • Unlimited and configurable parallelism with low overhead, supporting millions of simultaneous or short-duration workers, including micro-batches
  • Row and column high-performance copyingcomparing and mapping facilities for out-of-box and custom workers
  • Statically typed, high throughput dataflow with highly tuned workers
  • Bulk Insert for selected database providers

  • Wraps and extends .NET database providers to simplify writing database agnostic code
  • Dedicated MariaDB, MySQL, PostgreSQL®, SQLite and SQL Server® providers
  • ODBC provider for other databases
  • Local transactions across multiple workers
  • Supports many database specific data types (SqlDateTime, NpgsqlDate, …) end to end, avoiding conversion issues and reducing coding
  • Delimited and Fixed Format flat files
  • XLSX spreadsheet files
  • Extensible to other formats

Broad Data Source Support

Familiar ETL Concepts

  • Control flow with start constraintsgrouping, hierarchical structure and highly parallel applications
  • 50 dataflow workers provide extensive row-by-row processing capabilities
  • Read and write data sources, cleansecombine and transform data
  • Divide and conquer – implement requirements using many small (and reusable) parts
  • Effective debugging with breakpoints on worker and port state changes, and inspecting and editing rows in flight
  • Highly capable (as well as replaceable) logging and configuration systems
  • Automatically track and aggregate dataflow error row counts and log error row contents
  • Common tasks require very little programming, more akin to setting a configuration
  • Familiar concepts, project templates, a concise API and extensive documentation makes it easy to learn

  • NuGet package provides simple integration
  • Compose and encapsulate existing workers into new (control flow and dataflow) reusable workers
  • Highly flexible workers that optionally accept code snippets (lambdas), e.g. to perform a greater-than join instead of the default equi-join, or specifying ordering columns, etc.
  • .NET code-based programming and the actionETL architecture handles complexity very well
  • Perform ETL testing with any .NET test framework
  • Take full advantage of Visual Studio® or other .NET development environments for refactoring, source control, CI/CD, etc.

Modern Application Development

Unique Strengths

  • Merged Dataflow and Control flow functionality (including constraints), which reduces code complexity and data staging
  • Reuse and combine column schemas (groups of columns) to minimize dataflow bugs, maintenance effort and code size
  • Single programming model for both using and extending the library, including for constraints, custom workers etc., simplifying creating reusable custom functionality
  • Distinct recoverable vs. unrecoverable failures, minimizing start constraint bugs

Examples

Easy to Learn

Build your ETL application using the included 68 workers, which provide control flow and dataflow functionality, and handle both simple and highly complex scenarios with ease. The database workers generally work with all supported providers, making it easy to implement as well as port between databases.

This small application moves a file and groups three workers that execute an external command, read a CSV file and insert the records in a database.

actionETL worker hierarchy

The architecture also makes it easy to create new, custom workers, that can be used and reused, same as the out-of-box workers.

With excellent reusability and composability, actionETL required 23 times less C# code (9kB) to create a high performance and reusable custom Slowly Changing Dimension (SCD) worker (fully included in documentation) vs. similar functionality implemented in one commonly used traditional ETL tool (209kB).

23 Times Less Code

Excels with Large and Complex Requirements

In this comprehensive Process Incoming Files Example, a custom worker provides standardized and reliable processing, archiving and logging of incoming files, for any file type.

Platforms

actionETL runs on Windows .NET Framework 4.6.1+, and is also being ported to .NET Standard / .NET Core to run cross-platform, followed by .NET 5.

With .NET Standard in place, we’ll also start adding cloud specific features.