Core Concepts

This section introduces the fundamental concepts behind pipeline-flow, particularly aimed at new users and developers. It provides an overview of the core components of the framework, such as data pipelines, concurrency, plugins, and pipeline orchestration, to help you understand how the framework works.

Please refer to the following topics to get started:

• Data Pipelines
  • What is a Data Pipeline?
  • Types of Data Pipelines
  • Next Steps
• Concurrency Basics
  • Threading in Python
  • Async Programming
  • Why Choose Asynchronous Programming Over Threads?
  • Concurrency in pipeline-flow
  • Next Steps
• Plugins
  • What exactly is a Plugin?
  • Why use Plugins?
  • Plugin Registry
  • Types of Plugins
  • Next Steps
• Pipeline Orchestration
  • Pipeline phases: From Raw Data to Insights
  • Execution Flow
  • Next Steps