Skip to main content

Architecture

Data Formulator follows a modular microservices architecture designed for scalability and reliability. The system consists of several core components that work together to process and transform data efficiently.

The primary architecture includes a data ingestion layer that handles incoming data streams from various sources, a processing engine built on Apache Kafka for real-time data transformation, and a storage layer utilizing both relational and NoSQL databases depending on data types. The system employs a REST API gateway for external communications and includes built-in monitoring and alerting capabilities.

Key architectural components include:

  • Frontend web interface for configuration management

  • Backend processing services written in Java and Python

  • Message queue system for asynchronous processing

  • Distributed caching layer using Redis

  • Load balancer for high availability

The system supports horizontal scaling and can be deployed across multiple nodes in both on-premises and cloud environments.