Collection Tasks
Definition
Collection tasks are used to configure and execute metadata collection jobs. They support automatically extracting structure information, attribute information, and change information from objects such as data sources, databases, tables, and fields, and saving the collection results to the Metadata Management module. These results provide the basic data for metadata search, asset inventory, lineage analysis, and governance evaluation.
Core Capabilities:
- ✅ Task management: supports adding, editing, deleting, enabling, disabling, and running collection tasks once;
- 🗂️ Data source selection: supports selecting connected data sources as metadata collection objects;
- 🔍 Collection scope configuration: supports configuring the collection scope at database, table, and field granularity to avoid repeatedly collecting irrelevant objects;
- ⏱️ Scheduled execution: supports configuring schedule cycles based on business needs to regularly synchronize metadata changes;
- 📋 Execution records: records the execution time, execution status, collection results, and exception information of each collection;
- 🔗 Governance linkage: collection results can be used in governance scenarios such as metadata search, data asset identification, and field standard mapping.
Notes
Before running a collection task, ensure that the data source connection is normal and that the required metadata read permissions have been granted. For data sources with many tables or complex structures, configure the collection scope and schedule cycle properly to avoid extra pressure on the source database.
Scope
For data governance personnel, data administrators, and platform O&M personnel, this module is used to centrally configure metadata collection tasks and automatically synchronize data source structures and field information. It applies to governance scenarios such as data asset inventory, data catalog construction, metadata retrieval, data lineage analysis, and data standard mapping.
1. Collection Task Page
Click Data Governance -> Metadata Management -> Collection Tasks to enter the collection task page. The page displays the collection task list, usually including fields such as task name, data source type, data source name, collection scope, task status, execution policy, latest execution time, and latest execution status.
2. Add Collection Task
Click the Add button on the page. In the pop-up dialog, fill in the basic collection task information, select the data source to collect, and configure the collection scope, execution policy, and schedule cycle. The scheduler can be Quartz (lightweight) or Dolphinscheduler (enterprise-level). After confirming the information, click OK to add the collection task.
3. Edit Collection Task
Select the collection task to adjust, click Edit in the operation column, modify information such as task name, collection scope, and schedule cycle in the pop-up dialog, and click OK to save the changes.
4. Delete Collection Task
Select the collection task to delete, click Delete in the operation column, and click OK in the confirmation prompt. After deletion, the collection task will no longer be executed.
5. Enable/Disable Collection Task
Select the collection task to enable or disable, and click the switch in the task status column to complete the enable or disable operation. A disabled task will not be automatically executed according to the schedule cycle.
6. Run Once
Select the collection task to run immediately and click Run Once in the operation column. The system immediately starts a metadata collection, and updates the latest execution status and collection result after the execution is complete.
7. View Collection Instances
Select the collection task whose execution status you want to view, and click Execution Records in the operation column to view the historical execution list of the task. Execution records can be used to troubleshoot collection failures, confirm the collected data volume, and track collection changes.
