Cleansing Rules
Definition
This module is used to configure and manage various data cleansing rules. It aims to improve data accuracy, completeness, consistency, uniqueness, validity, and timeliness, ensuring that data reaches high quality standards before being opened through the middle-platform.
Covered Core Cleansing Types:
- 🎯 Accuracy correction: handles incorrect or inconsistent values in data, such as abnormal value correction and format standardization.
- ✅ Completeness repair: fills missing data, deletes invalid records, and ensures data completeness.
- 🔄 Consistency correction: unifies units, formats, value ranges, and more, eliminating differences across fields or tables.
- 🔑 Uniqueness maintenance: performs deduplication, merging, and other operations to avoid duplicate data.
- 🛡️ Validity processing: replaces illegal values, removes dirty data, and ensures legal field values.
- ⏱️ Timeliness adjustment: corrects abnormal time fields, fills missing time values, and improves time data accuracy.
Management Function Description:
This module focuses on improving data accuracy, completeness, consistency, uniqueness, validity, and timeliness. It covers many repair strategies, including numeric boundary adjustment, field type conversion, format standardization, field completion and truncation, null value filling, duplicate record processing, illegal value replacement, sensitive word filtering, and time correction. Through configurable identification and correction of dirty data, it unifies data formats, standardizes value ranges, ensures reasonable logic and complete structure, and provides a reliable foundation for subsequent Data Governance and analytics applications.
- Establish a cleansing rule classification system that supports level-1 and level-2 categories.
- Basic information can be configured, including but not limited to:
- Code
- Name
- Description
- Use cases
- Examples
Notes
The system presets dozens of cleansing rules. If you need to add rules, each rule code must stay consistent with the cleansing logic already implemented in the backend. Rule configuration takes effect only when the code and logic match in both directions. Add, delete, update, and query data must be inserted into the table manually.
Scope
Cleansing Rules Page
Click Cleansing Rules under Basic Management -> Rule Management to open the Cleansing Rules page. The page uses a left-right layout: the left side displays quality dimensions in a tree structure, and the right side displays the rule list, including rule name, rule description, rule level, rule type, and other fields.
