Tag: Data Engineering
Running dbt Rescue Rebuild in Production: Operational Playbooks, Failure Models, and Recovery Patterns
Go beyond the setup and into real-world execution. Learn how we run dbt rescue rebuilds in production: scoping dependencies, managing warehouse contention, handling incremental models, and recovering from outages with precision, without introducing new risks to pipeline stability.
The Rescue dbt_rerun Deployment: Rebuilding Changed and Broken Models Without Disrupting Production
Keeping production data correct after a dbt change is harder than it looks. Learn how we introduced a dedicated rescue deployment to rebuild exactly what’s needed and when it’s needed, bringing consistency back to production data without costly full reruns or pipeline disruptions.
Why Data Teams Struggle Without Separate Dev and Prod Environments
When development and production share the same data environment, even small changes can trigger costly outages. This article explains why separating dev and prod is foundational for reliable analytics, and how teams can do it without overengineering or blowing the budget.
SAP Data Ingestion with Python: A Technical Breakdown of Using the SAP RFC Protocol
Streamline SAP data integration with Python by leveraging the RFC protocol. This interview with the lead engineer of a new SAP RFC Connector explores the challenges of large-scale data extraction and explains how a C++ integration improves stability, speed, and reliability for modern data workflows.
Getting to Your First Flow Run: Prefect Worker & Deployment Setup
Run your first data ingestion workflow with Prefect, Docker, and Kubernetes. This guide walks through containerized flow execution, Prefect worker deployment, and clean deployment configs, laying the foundation for a scalable, maintainable orchestration layer.
Roles in the Context of the Analytics Workflow
Code-based analytics workflows offer a unique advantage - the ability to combine robust collaboration with strict governance. While technically straightforward to implement, this approach only reaches its full potential when aligned with well-adapted business and analytics processes.
dlt and Prefect, a Great Combo for Streamlined Data Ingestion Pipelines
Streamline your data ingestion pipelines with dlt and Prefect. This article explores how combining these powerful open-source tools enables scalable, efficient, and production-ready data workflows. Learn best practices, key features, and real-world insights to simplify data engineering.