Data Factory is the component that moves data into the lake in Fabric. Regular transfers from ERP, database, file and API sources are defined here.
01. Two Building Blocks
Dataflows let you transform data through a visual interface and feel familiar to anyone who knows Power Query. Pipelines are the orchestration layer that sequences steps and supports conditions and loops.
02. Incremental Loading
Pulling an entire table every night is fine on small data and burns capacity on millions of rows. Fetching only new records by modification date or an increasing key should be the standard.
03. On-Premises Sources
Reaching a SQL or ERP database on your own server requires a gateway. The continuity of the machine hosting the gateway is the continuity of the whole pipeline.
04. Error Handling
What happens when a step fails must be defined: retry, skip the next step, or notify whom. Overnight jobs that stop silently have people reading reports on stale data in the morning.
05. Environment Separation
Development and production should live in separate workspaces, with connection details managed through parameters. Otherwise every publish requires manual correction.
06. Documentation
Which pipeline feeds which table at what frequency should be written down. It's the only practical way to find affected reports when a data source changes.