Dataflow Node

Overview

Create a Dataflow Node

In the Offline Dev task editor, drag a Dataflow node from the left panel to the canvas. Select the node, then click Edit to open the operator editor.

Operator Lineage Analysis

Click the lineage icon on any operator, then select a field to view its upstream operator.

Execution Engine

Version 8.3 and later use the Gluten engine, an enhanced Spark engine, to run Dataflow by default and improve performance.

Detailed Description

Dataflow Node includes the following operators. Refer to the corresponding documents as needed.

Dataset Operators

Operator	Reference
Input Dataset	Input Dataset
Output Dataset	Output Dataset
Database Input (Advanced)	Database Input (Advanced)
Database Output (Advanced)	Database Output (Advanced)

Column Editing Operators

Operator	Reference
Add Calculated Column	Add Calculated Column
Merge Columns	Merge Columns
Group Aggregate	Group Aggregate
Select Columns	Select Columns
Pivot Rows to Columns	Pivot Rows to Columns
Unpivot Columns to Rows	Unpivot Columns to Rows

Data Editing Operators

Operator	Reference
Filter Data Rows	Filter Data Rows
Deduplicate	Deduplicate
Replace Values	Replace Values
Replace Null Values	Replace Null Values

Dataset Combination

Operator	Reference
Append Rows	Append Rows
Join Data	Join Data

Advanced Calculation

SQL operator: SQL Input