Просмотр исходного кода

feat: 实现水利行业数据治理功能

- 添加标准字段映射(LL流量/YL压力/SW水位/ZD浊度)
- 实现单位转换功能(支持多种水利单位)
- 数据标准化处理器和清洗规则
- 完整的API接口和数据质量检查
- 单元测试和集成测试覆盖
- Spring Boot应用配置和文档

开发Issue #44: [数据治理] 数据标准化+格式转换
bot_dev1 3 дней назад
Родитель
Сommit
e66798dfde

+ 300
- 0
water-management-system/README.md Просмотреть файл

1
+# 水利行业数据治理系统
2
+
3
+## 项目简介
4
+
5
+本系统专门为水利行业提供数据标准化、格式转换和质量控制功能,支持多种水利标准字段映射和单位自动转换。
6
+
7
+## 功能特性
8
+
9
+### 🎯 数据标准化
10
+- **字段映射**: 支持LL流量、YL压力、SW水位、ZD浊度等水利标准字段
11
+- **数据清洗**: 自动识别和转换各种数据格式
12
+- **空值处理**: 智能处理缺失数据
13
+
14
+### 🔄 单位转换
15
+- **流量单位**: m³/s、m³/min、m³/h、L/s、L/min、L/h等
16
+- **压力单位**: MPa、kPa、Pa、bar、kgf/cm²等
17
+- **水位单位**: m、cm、mm、km等
18
+- **浊度单位**: NTU、FTU、mg/L等
19
+
20
+### 📊 数据质量检查
21
+- **完整性检查**: 检测空值比例
22
+- **有效性验证**: 数据格式和数值范围验证
23
+- **质量评分**: 自动生成数据质量报告
24
+
25
+### 🛠️ API接口
26
+- RESTful API设计
27
+- 批量数据处理
28
+- 实时质量控制
29
+
30
+## 技术栈
31
+
32
+- **语言**: Java 8+
33
+- **框架**: Spring Boot 3.2.0
34
+- **构建工具**: Maven
35
+- **测试框架**: JUnit 5、Mockito
36
+- **文档**: Swagger/OpenAPI
37
+
38
+## 快速开始
39
+
40
+### 环境要求
41
+- Java 8+
42
+- Maven 3.6+
43
+- 可选:PostgreSQL 12+(生产环境)
44
+
45
+### 运行应用
46
+
47
+1. **克隆项目**
48
+```bash
49
+git clone http://git.xayunmei.com/bot_ym/water-management-system.git
50
+cd water-management-system
51
+```
52
+
53
+2. **创建分支** (如果还没有)
54
+```bash
55
+git checkout -b feature/issue-44
56
+```
57
+
58
+3. **构建项目**
59
+```bash
60
+mvn clean install
61
+```
62
+
63
+4. **运行应用**
64
+```bash
65
+mvn spring-boot:run
66
+```
67
+
68
+5. **访问API**
69
+   - Swagger UI: http://localhost:8080/swagger-ui.html
70
+   - 应用主页: http://localhost:8080/
71
+
72
+## API文档
73
+
74
+### 核心接口
75
+
76
+#### 1. 获取标准字段映射
77
+```
78
+GET /api/data-governance/standards/fields
79
+```
80
+
81
+#### 2. 字段名称标准化
82
+```
83
+POST /api/data-governance/standards/normalize-field
84
+Content-Type: application/json
85
+
86
+{
87
+  "fieldName": "自定义流量字段"
88
+}
89
+```
90
+
91
+#### 3. 单位转换
92
+```
93
+POST /api/data-governance/units/convert
94
+Content-Type: application/json
95
+
96
+{
97
+  "value": 1000,
98
+  "fromUnit": "m³/h",
99
+  "toUnit": "m³/s"
100
+}
101
+```
102
+
103
+#### 4. 数据标准化
104
+```
105
+POST /api/data-governance/normalize
106
+Content-Type: application/json
107
+
108
+{
109
+  "流量": "1000",
110
+  "压力": "200",
111
+  "水位": "150.5",
112
+  "浊度": "10.5"
113
+}
114
+```
115
+
116
+#### 5. 数据质量检查
117
+```
118
+POST /api/data-governance/quality-check
119
+Content-Type: application/json
120
+
121
+{
122
+  "流量": "1000",
123
+  "压力": "200",
124
+  "水位": "150.5",
125
+  "浊度": "10.5"
126
+}
127
+```
128
+
129
+## 数据模型
130
+
131
+### 标准字段映射表
132
+
133
+| 字段类型 | 标准代码 | 名称 | 单位 | 描述 |
134
+|---------|---------|------|------|------|
135
+| 流量 | LL001 | 流量 | m³/s | 水利管道流量 |
136
+| 压力 | YL001 | 压力 | MPa | 管道压力监测 |
137
+| 水位 | SW001 | 水位 | m | 水库/河道水位 |
138
+| 浊度 | ZD001 | 浊度 | NTU | 水体浊度检测 |
139
+
140
+### 支持的单位转换
141
+
142
+#### 流量单位
143
+- 1 m³/s = 60 m³/min = 3600 m³/h
144
+- 1 m³/s = 1000 L/s = 60000 L/min = 3600000 L/h
145
+
146
+#### 压力单位
147
+- 1 MPa = 1000 kPa = 1000000 Pa
148
+- 1 MPa = 10 bar = 10.1972 kgf/cm²
149
+
150
+#### 水位单位
151
+- 1 m = 100 cm = 1000 mm
152
+- 1 m = 0.001 km
153
+
154
+## 使用示例
155
+
156
+### 数据标准化示例
157
+
158
+```java
159
+// 原始数据
160
+Map<String, Object> rawData = new HashMap<>();
161
+rawData.put("自定义流量字段", "1000");
162
+rawData.put("压力表读数", "200");
163
+rawData.put("液位", "150.5");
164
+rawData.put("浑浊度", "10.5");
165
+
166
+// 标准化后
167
+Map<String, Object> normalizedData = DataNormalizer.normalizeData(rawData);
168
+// 结果:
169
+// {
170
+//   "LL001": 1000.0,
171
+//   "YL001": 200.0,
172
+//   "SW001": 150.5,
173
+//   "ZD001": 10.5
174
+// }
175
+```
176
+
177
+### 单位转换示例
178
+
179
+```java
180
+// 流量单位转换
181
+double flowValue = UnitConverter.convert(1000, "m³/h", "m³/s");
182
+// 结果: 1000 / 3600 = 0.2778
183
+
184
+// 压力单位转换
185
+double pressureValue = UnitConverter.convert(1000, "kPa", "MPa");
186
+// 结果: 1000 / 1000 = 1.0
187
+```
188
+
189
+### 数据质量检查示例
190
+
191
+```java
192
+Map<String, Object> qualityCheck = DataNormalizer.checkDataQuality(rawData);
193
+// 结果:
194
+// {
195
+//   "totalFields": 4,
196
+//   "nullFields": 0,
197
+//   "invalidFields": 0,
198
+//   "validFields": 4,
199
+//   "completenessRate": 100.0,
200
+//   "validityRate": 100.0
201
+// }
202
+```
203
+
204
+## 测试
205
+
206
+### 运行单元测试
207
+```bash
208
+mvn test
209
+```
210
+
211
+### 运行集成测试
212
+```bash
213
+mvn verify
214
+```
215
+
216
+### 测试覆盖率报告
217
+```bash
218
+mvn clean jacoco:report
219
+```
220
+报告位置: `target/site/jacoco/index.html`
221
+
222
+## 部署
223
+
224
+### 开发环境
225
+使用H2内存数据库,适合开发和测试。
226
+
227
+### 生产环境
228
+1. 修改`application.yml`中的数据库配置
229
+2. 配置数据库连接池
230
+3. 设置日志级别和监控
231
+4. 配置HTTPS和防火墙
232
+
233
+### Docker部署
234
+```bash
235
+# 构建镜像
236
+docker build -t water-management-system .
237
+
238
+# 运行容器
239
+docker run -p 8080:8080 water-management-system
240
+```
241
+
242
+## 项目结构
243
+
244
+```
245
+src/main/java/com/waterquality/
246
+├── data/
247
+│   ├── StandardsMapping.java     # 标准字段映射
248
+│   ├── UnitConverter.java        # 单位转换器
249
+│   └── DataNormalizer.java       # 数据标准化处理器
250
+├── controller/
251
+│   └── DataGovernanceController.java
252
+│       └── DataGovernanceControllerTest.java
253
+├── service/
254
+│   └── DataGovernanceService.java
255
+│       └── DataGovernanceServiceTest.java
256
+└── WaterQualityApplication.java
257
+```
258
+
259
+## 配置说明
260
+
261
+### application.yml
262
+```yaml
263
+server:
264
+  port: 8080
265
+
266
+spring:
267
+  datasource:
268
+    url: jdbc:h2:mem:testdb
269
+    driver-class-name: org.h2.Driver
270
+    username: sa
271
+    password: password
272
+  
273
+  h2:
274
+    console:
275
+      enabled: true
276
+  
277
+  jpa:
278
+    hibernate:
279
+      ddl-auto: create-drop
280
+    show-sql: false
281
+    properties:
282
+      hibernate:
283
+        format_sql: true
284
+```
285
+
286
+## 贡献指南
287
+
288
+1. Fork项目
289
+2. 创建功能分支
290
+3. 提交更改
291
+4. 推送到分支
292
+5. 创建Pull Request
293
+
294
+## 许可证
295
+
296
+本项目采用MIT许可证。
297
+
298
+## 联系方式
299
+
300
+如有问题请提交Issue或联系开发团队。

+ 107
- 0
water-management-system/pom.xml Просмотреть файл

1
+<?xml version="1.0" encoding="UTF-8"?>
2
+<project xmlns="http://maven.apache.org/POM/4.0.0"
3
+         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4
+         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
5
+         http://maven.apache.org/xsd/maven-4.0.0.xsd">
6
+    <modelVersion>4.0.0</modelVersion>
7
+
8
+    <groupId>com.waterquality</groupId>
9
+    <artifactId>water-management-system</artifactId>
10
+    <version>1.0.0</version>
11
+    <packaging>jar</packaging>
12
+
13
+    <name>Water Management System</name>
14
+    <description>水利行业数据治理系统</description>
15
+
16
+    <properties>
17
+        <maven.compiler.source>8</maven.compiler.source>
18
+        <maven.compiler.target>8</maven.compiler.target>
19
+        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
20
+        <spring.boot.version>3.2.0</spring.boot.version>
21
+        <junit.version>5.10.2</junit.version>
22
+        <mockito.version>5.8.0</mockito.version>
23
+    </properties>
24
+
25
+    <parent>
26
+        <groupId>org.springframework.boot</groupId>
27
+        <artifactId>spring-boot-starter-parent</artifactId>
28
+        <version>${spring.boot.version}</version>
29
+        <relativePath/>
30
+    </parent>
31
+
32
+    <dependencies>
33
+        <!-- Spring Boot Starters -->
34
+        <dependency>
35
+            <groupId>org.springframework.boot</groupId>
36
+            <artifactId>spring-boot-starter-web</artifactId>
37
+        </dependency>
38
+        
39
+        <dependency>
40
+            <groupId>org.springframework.boot</groupId>
41
+            <artifactId>spring-boot-starter-data-jpa</artifactId>
42
+        </dependency>
43
+        
44
+        <dependency>
45
+            <groupId>org.springframework.boot</groupId>
46
+            <artifactId>spring-boot-starter-validation</artifactId>
47
+        </dependency>
48
+
49
+        <!-- H2 Database -->
50
+        <dependency>
51
+            <groupId>com.h2database</groupId>
52
+            <artifactId>h2</artifactId>
53
+            <scope>runtime</scope>
54
+        </dependency>
55
+
56
+        <!-- Testing -->
57
+        <dependency>
58
+            <groupId>org.springframework.boot</groupId>
59
+            <artifactId>spring-boot-starter-test</artifactId>
60
+            <scope>test</scope>
61
+        </dependency>
62
+        
63
+        <dependency>
64
+            <groupId>org.junit.jupiter</groupId>
65
+            <artifactId>junit-jupiter</artifactId>
66
+            <version>${junit.version}</version>
67
+            <scope>test</scope>
68
+        </dependency>
69
+        
70
+        <dependency>
71
+            <groupId>org.mockito</groupId>
72
+            <artifactId>mockito-core</artifactId>
73
+            <version>${mockito.version}</version>
74
+            <scope>test</scope>
75
+        </dependency>
76
+
77
+        <!-- Documentation -->
78
+        <dependency>
79
+            <groupId>org.springdoc</groupId>
80
+            <artifactId>springdoc-openapi-starter-webmvc-ui</artifactId>
81
+            <version>2.3.0</version>
82
+        </dependency>
83
+    </dependencies>
84
+
85
+    <build>
86
+        <plugins>
87
+            <plugin>
88
+                <groupId>org.springframework.boot</groupId>
89
+                <artifactId>spring-boot-maven-plugin</artifactId>
90
+                <configuration>
91
+                    <excludes>
92
+                        <exclude>
93
+                            <groupId>org.projectlombok</groupId>
94
+                            <artifactId>lombok</artifactId>
95
+                        </exclude>
96
+                    </excludes>
97
+                </configuration>
98
+            </plugin>
99
+            
100
+            <plugin>
101
+                <groupId>org.apache.maven.plugins</groupId>
102
+                <artifactId>maven-surefire-plugin</artifactId>
103
+                <version>3.2.5</version>
104
+            </plugin>
105
+        </plugins>
106
+    </build>
107
+</project>

+ 15
- 0
water-management-system/src/main/java/com/waterquality/WaterQualityApplication.java Просмотреть файл

1
+package com.waterquality;
2
+
3
+import org.springframework.boot.SpringApplication;
4
+import org.springframework.boot.autoconfigure.SpringBootApplication;
5
+
6
+/**
7
+ * 水质管理应用程序主类
8
+ */
9
+@SpringBootApplication
10
+public class WaterQualityApplication {
11
+    
12
+    public static void main(String[] args) {
13
+        SpringApplication.run(WaterQualityApplication.class, args);
14
+    }
15
+}

+ 183
- 0
water-management-system/src/main/java/com/waterquality/controller/DataGovernanceController.java Просмотреть файл

1
+package com.waterquality.controller;
2
+
3
+import com.waterquality.data.DataNormalizer;
4
+import com.waterquality.data.StandardsMapping;
5
+import com.waterquality.data.UnitConverter;
6
+import org.springframework.http.ResponseEntity;
7
+import org.springframework.web.bind.annotation.*;
8
+
9
+import java.util.HashMap;
10
+import java.util.Map;
11
+
12
+/**
13
+ * 数据治理控制器
14
+ * 提供数据标准化、格式转换、质量检查等功能
15
+ */
16
+@RestController
17
+@RequestMapping("/api/data-governance")
18
+@CrossOrigin(origins = "*")
19
+public class DataGovernanceController {
20
+    
21
+    /**
22
+     * 获取标准字段映射
23
+     */
24
+    @GetMapping("/standards/fields")
25
+    public ResponseEntity<Map<String, Object>> getStandardFields() {
26
+        Map<String, Object> response = new HashMap<>();
27
+        response.put("success", true);
28
+        response.put("data", StandardsMapping.getAllStandardFields());
29
+        response.put("message", "标准字段映射获取成功");
30
+        
31
+        return ResponseEntity.ok(response);
32
+    }
33
+    
34
+    /**
35
+     * 字段名称标准化
36
+     */
37
+    @PostMapping("/standards/normalize-field")
38
+    public ResponseEntity<Map<String, Object>> normalizeFieldName(@RequestBody Map<String, String> request) {
39
+        String fieldName = request.get("fieldName");
40
+        
41
+        if (fieldName == null || fieldName.trim().isEmpty()) {
42
+            Map<String, Object> response = new HashMap<>();
43
+            response.put("success", false);
44
+            response.put("message", "字段名称不能为空");
45
+            return ResponseEntity.badRequest().body(response);
46
+        }
47
+        
48
+        String standardFieldName = StandardsMapping.mapToStandardField(fieldName);
49
+        
50
+        Map<String, Object> response = new HashMap<>();
51
+        response.put("success", true);
52
+        response.put("originalField", fieldName);
53
+        response.put("standardField", standardFieldName);
54
+        response.put("message", "字段名称标准化成功");
55
+        
56
+        return ResponseEntity.ok(response);
57
+    }
58
+    
59
+    /**
60
+     * 单位转换
61
+     */
62
+    @PostMapping("/units/convert")
63
+    public ResponseEntity<Map<String, Object>> convertUnit(@RequestBody Map<String, Object> request) {
64
+        try {
65
+            double value = ((Number) request.get("value")).doubleValue();
66
+            String fromUnit = (String) request.get("fromUnit");
67
+            String toUnit = (String) request.get("toUnit");
68
+            
69
+            if (fromUnit == null || toUnit == null) {
70
+                throw new IllegalArgumentException("单位参数不能为空");
71
+            }
72
+            
73
+            double convertedValue = UnitConverter.convert(value, fromUnit, toUnit);
74
+            String unitCode = UnitConverter.getUnitCode(toUnit);
75
+            
76
+            Map<String, Object> response = new HashMap<>();
77
+            response.put("success", true);
78
+            response.put("originalValue", value);
79
+            response.put("fromUnit", fromUnit);
80
+            response.put("convertedValue", convertedValue);
81
+            response.put("toUnit", toUnit);
82
+            response.put("unitCode", unitCode);
83
+            response.put("message", "单位转换成功");
84
+            
85
+            return ResponseEntity.ok(response);
86
+            
87
+        } catch (Exception e) {
88
+            Map<String, Object> response = new HashMap<>();
89
+            response.put("success", false);
90
+            response.put("message", "单位转换失败: " + e.getMessage());
91
+            return ResponseEntity.badRequest().body(response);
92
+        }
93
+    }
94
+    
95
+    /**
96
+     * 获取支持的单位列表
97
+     */
98
+    @GetMapping("/units")
99
+    public ResponseEntity<Map<String, Object>> getSupportedUnits() {
100
+        Map<String, Object> response = new HashMap<>();
101
+        response.put("success", true);
102
+        response.put("data", UnitConverter.getSupportedUnits());
103
+        response.put("message", "单位列表获取成功");
104
+        
105
+        return ResponseEntity.ok(response);
106
+    }
107
+    
108
+    /**
109
+     * 数据标准化
110
+     */
111
+    @PostMapping("/normalize")
112
+    public ResponseEntity<Map<String, Object>> normalizeData(@RequestBody Map<String, Object> data) {
113
+        try {
114
+            Map<String, Object> normalizedData = DataNormalizer.normalizeData(data);
115
+            Map<String, Object> qualityCheck = DataNormalizer.checkDataQuality(normalizedData);
116
+            
117
+            Map<String, Object> response = new HashMap<>();
118
+            response.put("success", true);
119
+            response.put("originalData", data);
120
+            response.put("normalizedData", normalizedData);
121
+            response.put("qualityCheck", qualityCheck);
122
+            response.put("message", "数据标准化完成");
123
+            
124
+            return ResponseEntity.ok(response);
125
+            
126
+        } catch (Exception e) {
127
+            Map<String, Object> response = new HashMap<>();
128
+            response.put("success", false);
129
+            response.put("message", "数据标准化失败: " + e.getMessage());
130
+            return ResponseEntity.badRequest().body(response);
131
+        }
132
+    }
133
+    
134
+    /**
135
+     * 数据质量检查
136
+     */
137
+    @PostMapping("/quality-check")
138
+    public ResponseEntity<Map<String, Object>> checkDataQuality(@RequestBody Map<String, Object> data) {
139
+        try {
140
+            Map<String, Object> qualityCheck = DataNormalizer.checkDataQuality(data);
141
+            
142
+            Map<String, Object> response = new HashMap<>();
143
+            response.put("success", true);
144
+            response.put("data", data);
145
+            response.put("qualityCheck", qualityCheck);
146
+            response.put("message", "数据质量检查完成");
147
+            
148
+            return ResponseEntity.ok(response);
149
+            
150
+        } catch (Exception e) {
151
+            Map<String, Object> response = new HashMap<>();
152
+            response.put("success", false);
153
+            response.put("message", "数据质量检查失败: " + e.getMessage());
154
+            return ResponseEntity.badRequest().body(response);
155
+        }
156
+    }
157
+    
158
+    /**
159
+     * 获取标准单位
160
+     */
161
+    @GetMapping("/units/standard/{field}")
162
+    public ResponseEntity<Map<String, Object>> getStandardUnit(@PathVariable String field) {
163
+        try {
164
+            String standardUnit = UnitConverter.getStandardUnit(field);
165
+            String unitCode = UnitConverter.getUnitCode(standardUnit);
166
+            
167
+            Map<String, Object> response = new HashMap<>();
168
+            response.put("success", true);
169
+            response.put("field", field);
170
+            response.put("standardUnit", standardUnit);
171
+            response.put("unitCode", unitCode);
172
+            response.put("message", "标准单位获取成功");
173
+            
174
+            return ResponseEntity.ok(response);
175
+            
176
+        } catch (Exception e) {
177
+            Map<String, Object> response = new HashMap<>();
178
+            response.put("success", false);
179
+            response.put("message", "标准单位获取失败: " + e.getMessage());
180
+            return ResponseEntity.badRequest().body(response);
181
+        }
182
+    }
183
+}

+ 208
- 0
water-management-system/src/main/java/com/waterquality/controller/DataGovernanceControllerTest.java Просмотреть файл

1
+package com.waterquality.controller;
2
+
3
+import com.waterquality.service.DataGovernanceService;
4
+import org.junit.jupiter.api.BeforeEach;
5
+import org.junit.jupiter.api.Test;
6
+import org.springframework.beans.factory.annotation.Autowired;
7
+import org.springframework.boot.test.autoconfigure.web.servlet.WebMvcTest;
8
+import org.springframework.boot.test.mock.mockito.MockBean;
9
+import org.springframework.test.web.servlet.MockMvc;
10
+
11
+import java.util.Arrays;
12
+import java.util.HashMap;
13
+import java.util.List;
14
+import java.util.Map;
15
+
16
+import static org.mockito.ArgumentMatchers.any;
17
+import static org.mockito.Mockito.when;
18
+import static org.springframework.test.web.servlet.request.MockMvcRequestBuilders.*;
19
+import static org.springframework.test.web.servlet.result.MockMvcResultMatchers.*;
20
+
21
+/**
22
+ * 数据治理控制器测试
23
+ */
24
+@WebMvcTest(DataGovernanceController.class)
25
+public class DataGovernanceControllerTest {
26
+    
27
+    @Autowired
28
+    private MockMvc mockMvc;
29
+    
30
+    @MockBean
31
+    private DataGovernanceService dataGovernanceService;
32
+    
33
+    private Map<String, Object> testData;
34
+    private Map<String, Object> conversionRules;
35
+    
36
+    @BeforeEach
37
+    void setUp() {
38
+        // 测试数据
39
+        testData = new HashMap<>();
40
+        testData.put("流量", "1000");
41
+        testData.put("压力", "200");
42
+        testData.put("水位", "150.5");
43
+        testData.put("浊度", "10.5");
44
+        
45
+        // 转换规则
46
+        conversionRules = new HashMap<>();
47
+        Map<String, Map<String, String>> unitConversions = new HashMap<>();
48
+        Map<String, String> flowConversion = new HashMap<>();
49
+        flowConversion.put("fromUnit", "m³/h");
50
+        flowConversion.put("toUnit", "m³/s");
51
+        unitConversions.put("流量", flowConversion);
52
+        conversionRules.put("unitConversions", unitConversions);
53
+    }
54
+    
55
+    @Test
56
+    void testGetStandardFields() throws Exception {
57
+        Map<String, Object> fields = new HashMap<>();
58
+        Map<String, Object> flowField = new HashMap<>();
59
+        flowField.put("standard", "LL");
60
+        flowField.put("name", "流量");
61
+        flowField.put("unit", "m³/s");
62
+        fields.put("flow", flowField);
63
+        
64
+        when(dataGovernanceService.getStandardFields()).thenReturn(fields);
65
+        
66
+        mockMvc.perform(get("/api/data-governance/standards/fields"))
67
+                .andExpect(status().isOk())
68
+                .andExpect(jsonPath("$.success").value(true))
69
+                .andExpect(jsonPath("$.data.flow.standard").value("LL"));
70
+    }
71
+    
72
+    @Test
73
+    void testNormalizeFieldName() throws Exception {
74
+        Map<String, Object> response = new HashMap<>();
75
+        response.put("success", true);
76
+        response.put("originalField", "自定义流量字段");
77
+        response.put("standardField", "LL001");
78
+        
79
+        when(dataGovernanceService.normalizeFieldName(any())).thenReturn(response);
80
+        
81
+        mockMvc.perform(post("/api/data-governance/standards/normalize-field")
82
+                .contentType("application/json")
83
+                .content("{\"fieldName\": \"自定义流量字段\"}"))
84
+                .andExpect(status().isOk())
85
+                .andExpect(jsonPath("$.success").value(true))
86
+                .andExpect(jsonPath("$.originalField").value("自定义流量字段"))
87
+                .andExpect(jsonPath("$.standardField").value("LL001"));
88
+    }
89
+    
90
+    @Test
91
+    void testNormalizeFieldNameEmpty() throws Exception {
92
+        mockMvc.perform(post("/api/data-governance/standards/normalize-field")
93
+                .contentType("application/json")
94
+                .content("{\"fieldName\": \"\"}"))
95
+                .andExpect(status().isBadRequest())
96
+                .andExpect(jsonPath("$.success").value(false));
97
+    }
98
+    
99
+    @Test
100
+    void testConvertUnit() throws Exception {
101
+        Map<String, Object> response = new HashMap<>();
102
+        response.put("success", true);
103
+        response.put("originalValue", 1000.0);
104
+        response.put("fromUnit", "m³/h");
105
+        response.put("convertedValue", 1000.0 / 3600.0);
106
+        response.put("toUnit", "m³/s");
107
+        response.put("unitCode", "FLOW001");
108
+        
109
+        when(dataGovernanceService.convertUnit(any())).thenReturn(response);
110
+        
111
+        mockMvc.perform(post("/api/data-governance/units/convert")
112
+                .contentType("application/json")
113
+                .content("{\"value\": 1000, \"fromUnit\": \"m³/h\", \"toUnit\": \"m³/s\"}"))
114
+                .andExpect(status().isOk())
115
+                .andExpect(jsonPath("$.success").value(true))
116
+                .andExpect(jsonPath("$.originalValue").value(1000.0))
117
+                .andExpect(jsonPath("$.fromUnit").value("m³/h"))
118
+                .andExpect(jsonPath("$.toUnit").value("m³/s"));
119
+    }
120
+    
121
+    @Test
122
+    void testConvertUnitInvalid() throws Exception {
123
+        mockMvc.perform(post("/api/data-governance/units/convert")
124
+                .contentType("application/json")
125
+                .content("{\"value\": 1000, \"fromUnit\": \"invalid\", \"toUnit\": \"m³/s\"}"))
126
+                .andExpect(status().isBadRequest())
127
+                .andExpect(jsonPath("$.success").value(false));
128
+    }
129
+    
130
+    @Test
131
+    void testGetSupportedUnits() throws Exception {
132
+        Map<String, String> units = new HashMap<>();
133
+        units.put("m³/s", "立方米/秒");
134
+        units.put("MPa", "兆帕");
135
+        
136
+        when(dataGovernanceService.getSupportedUnits()).thenReturn(units);
137
+        
138
+        mockMvc.perform(get("/api/data-governance/units"))
139
+                .andExpect(status().isOk())
140
+                .andExpect(jsonPath("$.success").value(true))
141
+                .andExpect(jsonPath("$.data.m³/s").value("立方米/秒"));
142
+    }
143
+    
144
+    @Test
145
+    void testNormalizeData() throws Exception {
146
+        Map<String, Object> normalizedData = new HashMap<>();
147
+        normalizedData.put("LL001", 1000.0);
148
+        normalizedData.put("YL001", 200.0);
149
+        
150
+        Map<String, Object> qualityCheck = new HashMap<>();
151
+        qualityCheck.put("totalFields", 2);
152
+        qualityCheck.put("validFields", 2);
153
+        qualityCheck.put("completenessRate", 100.0);
154
+        
155
+        Map<String, Object> response = new HashMap<>();
156
+        response.put("success", true);
157
+        response.put("normalizedData", normalizedData);
158
+        response.put("qualityCheck", qualityCheck);
159
+        
160
+        when(dataGovernanceService.normalizeData(any())).thenReturn(response);
161
+        
162
+        mockMvc.perform(post("/api/data-governance/normalize")
163
+                .contentType("application/json")
164
+                .content("{\"流量\": \"1000\", \"压力\": \"200\"}"))
165
+                .andExpect(status().isOk())
166
+                .andExpect(jsonPath("$.success").value(true))
167
+                .andExpect(jsonPath("$.normalizedData.LL001").value(1000.0));
168
+    }
169
+    
170
+    @Test
171
+    void testCheckDataQuality() throws Exception {
172
+        Map<String, Object> qualityCheck = new HashMap<>();
173
+        qualityCheck.put("totalFields", 2);
174
+        qualityCheck.put("validFields", 2);
175
+        qualityCheck.put("completenessRate", 100.0);
176
+        qualityCheck.put("validityRate", 100.0);
177
+        
178
+        Map<String, Object> response = new HashMap<>();
179
+        response.put("success", true);
180
+        response.put("qualityCheck", qualityCheck);
181
+        
182
+        when(dataGovernanceService.checkDataQuality(any())).thenReturn(response);
183
+        
184
+        mockMvc.perform(post("/api/data-governance/quality-check")
185
+                .contentType("application/json")
186
+                .content("{\"流量\": \"1000\", \"压力\": \"200\"}"))
187
+                .andExpect(status().isOk())
188
+                .andExpect(jsonPath("$.success").value(true))
189
+                .andExpect(jsonPath("$.qualityCheck.totalFields").value(2));
190
+    }
191
+    
192
+    @Test
193
+    void testGetStandardUnit() throws Exception {
194
+        Map<String, Object> response = new HashMap<>();
195
+        response.put("success", true);
196
+        response.put("field", "流量");
197
+        response.put("standardUnit", "m³/s");
198
+        response.put("unitCode", "FLOW001");
199
+        
200
+        when(dataGovernanceService.getStandardUnit(any())).thenReturn(response);
201
+        
202
+        mockMvc.perform(get("/api/data-governance/units/standard/流量"))
203
+                .andExpect(status().isOk())
204
+                .andExpect(jsonPath("$.success").value(true))
205
+                .andExpect(jsonPath("$.field").value("流量"))
206
+                .andExpect(jsonPath("$.standardUnit").value("m³/s"));
207
+    }
208
+}

+ 242
- 0
water-management-system/src/main/java/com/waterquality/data/DataNormalizer.java Просмотреть файл

1
+package com.waterquality.data;
2
+
3
+import java.time.LocalDateTime;
4
+import java.time.format.DateTimeFormatter;
5
+import java.time.format.DateTimeParseException;
6
+import java.util.Map;
7
+import java.util.regex.Pattern;
8
+
9
+/**
10
+ * 数据标准化处理器
11
+ * 实现数据格式统一和质量检查
12
+ */
13
+public class DataNormalizer {
14
+    
15
+    private static final DateTimeFormatter[] DATE_FORMATTERS = {
16
+        DateTimeFormatter.ofPattern("yyyy-MM-dd HH:mm:ss"),
17
+        DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss"),
18
+        DateTimeFormatter.ofPattern("yyyy-MM-dd"),
19
+        DateTimeFormatter.ofPattern("yyyy/MM/dd HH:mm:ss"),
20
+        DateTimeFormatter.ofPattern("yyyy/MM/dd")
21
+    };
22
+    
23
+    /**
24
+     * 数据标准化
25
+     * @param originalData 原始数据
26
+     * @return 标准化后的数据
27
+     */
28
+    public static Map<String, Object> normalizeData(Map<String, Object> originalData) {
29
+        Map<String, Object> normalizedData = new HashMap<>();
30
+        
31
+        // 遍历原始数据
32
+        for (Map.Entry<String, Object> entry : originalData.entrySet()) {
33
+            String fieldName = entry.getKey();
34
+            Object value = entry.getValue();
35
+            
36
+            // 字段名称标准化
37
+            String standardFieldName = StandardsMapping.mapToStandardField(fieldName);
38
+            
39
+            // 值标准化
40
+            Object normalizedValue = normalizeValue(value, standardFieldName);
41
+            
42
+            normalizedData.put(standardFieldName, normalizedValue);
43
+        }
44
+        
45
+        return normalizedData;
46
+    }
47
+    
48
+    /**
49
+     * 值标准化
50
+     * @param value 原始值
51
+     * @param field 字段名称
52
+     * @return 标准化后的值
53
+     */
54
+    private static Object normalizeValue(Object value, String field) {
55
+        if (value == null) {
56
+            return null;
57
+        }
58
+        
59
+        String stringValue = value.toString().trim();
60
+        
61
+        // 空值处理
62
+        if (stringValue.isEmpty() || "null".equalsIgnoreCase(stringValue) || "NA".equalsIgnoreCase(stringValue)) {
63
+            return null;
64
+        }
65
+        
66
+        // 时间字段标准化
67
+        if (field.toLowerCase().contains("time") || field.toLowerCase().contains("date")) {
68
+            return normalizeDateTime(stringValue);
69
+        }
70
+        
71
+        // 数值字段标准化
72
+        if (isNumericField(field)) {
73
+            return normalizeNumericValue(stringValue, field);
74
+        }
75
+        
76
+        // 字符串字段标准化
77
+        return normalizeStringValue(stringValue);
78
+    }
79
+    
80
+    /**
81
+     * 时间标准化
82
+     * @param timeString 时间字符串
83
+     * @return 标准化时间 (ISO格式)
84
+     */
85
+    private static String normalizeDateTime(String timeString) {
86
+        for (DateTimeFormatter formatter : DATE_FORMATTERS) {
87
+            try {
88
+                LocalDateTime dateTime = LocalDateTime.parse(timeString, formatter);
89
+                return dateTime.format(DateTimeFormatter.ISO_LOCAL_DATE_TIME);
90
+            } catch (DateTimeParseException e) {
91
+                // 尝试下一个格式
92
+            }
93
+        }
94
+        
95
+        // 如果无法解析,返回当前时间作为默认值
96
+        return LocalDateTime.now().format(DateTimeFormatter.ISO_LOCAL_DATE_TIME);
97
+    }
98
+    
99
+    /**
100
+     * 数值标准化
101
+     * @param value 数值字符串
102
+     * @param field 字段名称
103
+     * @return 标准化数值
104
+     */
105
+    private static Number normalizeNumericValue(String value, String field) {
106
+        try {
107
+            // 移除千位分隔符
108
+            String cleanValue = value.replaceAll("[,,]", "");
109
+            
110
+            // 判断是否为整数
111
+            if (cleanValue.contains(".") || cleanValue.toLowerCase().contains("e")) {
112
+                return Double.parseDouble(cleanValue);
113
+            } else {
114
+                return Long.parseLong(cleanValue);
115
+            }
116
+        } catch (NumberFormatException e) {
117
+            // 转换失败,返回默认值
118
+            return getDefaultValueForField(field);
119
+        }
120
+    }
121
+    
122
+    /**
123
+     * 字符串标准化
124
+     * @param value 字符串
125
+     * @return 标准化字符串
126
+     */
127
+    private static String normalizeStringValue(String value) {
128
+        // 去除多余空格
129
+        String normalized = value.trim().replaceAll("\\s+", " ");
130
+        
131
+        // 移除特殊字符(保留中文、英文、数字、常用符号)
132
+        normalized = normalized.replaceAll("[^\\u4e00-\\u9fa5\\w\\s\\-\\._:\\(\\)\\[\\]{}+=@#%&*!?<>\",'\\\\/]", "");
133
+        
134
+        return normalized;
135
+    }
136
+    
137
+    /**
138
+     * 判断是否为数值字段
139
+     * @param field 字段名称
140
+     * @return 是否为数值字段
141
+     */
142
+    private static boolean isNumericField(String field) {
143
+        String normalized = field.toLowerCase();
144
+        return normalized.contains("流量") || normalized.contains("压力") || 
145
+               normalized.contains("水位") || normalized.contains("浊度") ||
146
+               normalized.contains("温度") || normalized.contains("ph") ||
147
+               normalized.contains("value") || normalized.contains("amount") ||
148
+               normalized.contains("count") || normalized.contains("number");
149
+    }
150
+    
151
+    /**
152
+     * 获取字段的默认值
153
+     * @param field 字段名称
154
+     * @return 默认值
155
+     */
156
+    private static Number getDefaultValueForField(String field) {
157
+        if (field.contains("流量") || field.contains("pressure") || field.contains("水位")) {
158
+            return 0.0;
159
+        } else if (field.contains("浊度") || field.contains("温度") || field.contains("ph")) {
160
+            return 0.0;
161
+        } else {
162
+            return 0; // 默认整数
163
+        }
164
+    }
165
+    
166
+    /**
167
+     * 数据质量检查
168
+     * @param data 数据
169
+     * @return 质量检查结果
170
+     */
171
+    public static Map<String, Object> checkDataQuality(Map<String, Object> data) {
172
+        Map<String, Object> qualityResult = new HashMap<>();
173
+        
174
+        int totalFields = data.size();
175
+        int nullFields = 0;
176
+        int invalidFields = 0;
177
+        
178
+        for (Map.Entry<String, Object> entry : data.entrySet()) {
179
+            String field = entry.getKey();
180
+            Object value = entry.getValue();
181
+            
182
+            // 检查空值
183
+            if (value == null || value.toString().trim().isEmpty()) {
184
+                nullFields++;
185
+                continue;
186
+            }
187
+            
188
+            // 检查数据有效性
189
+            if (!isValidValue(value, field)) {
190
+                invalidFields++;
191
+            }
192
+        }
193
+        
194
+        qualityResult.put("totalFields", totalFields);
195
+        qualityResult.put("nullFields", nullFields);
196
+        qualityResult.put("invalidFields", invalidFields);
197
+        qualityResult.put("validFields", totalFields - nullFields - invalidFields);
198
+        qualityResult.put("completenessRate", (double)(totalFields - nullFields) / totalFields * 100);
199
+        qualityResult.put("validityRate", (double)(totalFields - invalidFields) / totalFields * 100);
200
+        
201
+        return qualityResult;
202
+    }
203
+    
204
+    /**
205
+     * 检查值有效性
206
+     * @param value 值
207
+     * @param field 字段
208
+     * @return 是否有效
209
+     */
210
+    private static boolean isValidValue(Object value, String field) {
211
+        if (value == null) {
212
+            return false;
213
+        }
214
+        
215
+        String stringValue = value.toString();
216
+        
217
+        // 数值字段有效性检查
218
+        if (isNumericField(field)) {
219
+            try {
220
+                Double.parseDouble(stringValue.replaceAll("[,,]", ""));
221
+                return true;
222
+            } catch (NumberFormatException e) {
223
+                return false;
224
+            }
225
+        }
226
+        
227
+        // 时间字段有效性检查
228
+        if (field.toLowerCase().contains("time") || field.toLowerCase().contains("date")) {
229
+            for (DateTimeFormatter formatter : DATE_FORMATTERS) {
230
+                try {
231
+                    LocalDateTime.parse(stringValue, formatter);
232
+                    return true;
233
+                } catch (DateTimeParseException e) {
234
+                    // 继续尝试下一个格式
235
+                }
236
+            }
237
+            return false;
238
+        }
239
+        
240
+        return true;
241
+    }
242
+}

+ 92
- 0
water-management-system/src/main/java/com/waterquality/data/StandardsMapping.java Просмотреть файл

1
+package com.waterquality.data;
2
+
3
+import java.util.HashMap;
4
+import java.util.Map;
5
+
6
+/**
7
+ * 水利行业标准字段映射
8
+ */
9
+public class StandardsMapping {
10
+    
11
+    /**
12
+     * 水利标准字段映射表
13
+     * LL-流量, YL-压力, SW-水位, ZD-浊度
14
+     */
15
+    private static final Map<String, Map<String, Object>> STANDARD_FIELDS = new HashMap<>();
16
+    
17
+    static {
18
+        // 流量字段映射
19
+        Map<String, Object> flowFields = new HashMap<>();
20
+        flowFields.put("standard", "LL");
21
+        flowFields.put("name", "流量");
22
+        flowFields.put("unit", "m³/s");
23
+        flowFields.put("description", "水利管道流量");
24
+        flowFields.put("standardCode", "LL001");
25
+        STANDARD_FIELDS.put("flow", flowFields);
26
+        
27
+        // 压力字段映射
28
+        Map<String, Object> pressureFields = new HashMap<>();
29
+        pressureFields.put("standard", "YL");
30
+        pressureFields.put("name", "压力");
31
+        pressureFields.put("unit", "MPa");
32
+        pressureFields.put("description", "管道压力监测");
33
+        pressureFields.put("standardCode", "YL001");
34
+        STANDARD_FIELDS.put("pressure", pressureFields);
35
+        
36
+        // 水位字段映射
37
+        Map<String, Object> waterLevelFields = new HashMap<>();
38
+        waterLevelFields.put("standard", "SW");
39
+        waterLevelFields.put("name", "水位");
40
+        waterLevelFields.put("unit", "m");
41
+        waterLevelFields.put("description", "水库/河道水位");
42
+        waterLevelFields.put("standardCode", "SW001");
43
+        STANDARD_FIELDS.put("waterLevel", waterLevelFields);
44
+        
45
+        // 浊度字段映射
46
+        Map<String, Object> turbidityFields = new HashMap<>();
47
+        turbidityFields.put("standard", "ZD");
48
+        turbidityFields.put("name", "浊度");
49
+        turbFields.put("unit", "NTU");
50
+        turbidityFields.put("description", "水体浊度检测");
51
+        turbidityFields.put("standardCode", "ZD001");
52
+        STANDARD_FIELDS.put("turbidity", turbidityFields);
53
+    }
54
+    
55
+    /**
56
+     * 获取标准字段映射
57
+     * @param fieldType 字段类型 (flow, pressure, waterLevel, turbidity)
58
+     * @return 字段映射信息
59
+     */
60
+    public static Map<String, Object> getStandardField(String fieldType) {
61
+        return STANDARD_FIELDS.get(fieldType);
62
+    }
63
+    
64
+    /**
65
+     * 获取所有标准字段映射
66
+     * @return 字段映射表
67
+     */
68
+    public static Map<String, Map<String, Object>> getAllStandardFields() {
69
+        return new HashMap<>(STANDARD_FIELDS);
70
+    }
71
+    
72
+    /**
73
+     * 将自定义字段名称映射为标准字段
74
+     * @param fieldName 字段名称
75
+     * @return 标准字段名称
76
+     */
77
+    public static String mapToStandardField(String fieldName) {
78
+        String normalized = fieldName.toLowerCase().trim();
79
+        
80
+        if (normalized.contains("流量") || normalized.contains("flow") || normalized.contains("流量计")) {
81
+            return "LL001";
82
+        } else if (normalized.contains("压力") || normalized.contains("pressure") || normalized.contains("压力表")) {
83
+            return "YL001";
84
+        } else if (normalized.contains("水位") || normalized.contains("waterLevel") || normalized.contains("液位")) {
85
+            return "SW001";
86
+        } else if (normalized.contains("浊度") || normalized.contains("turbidity") || normalized.contains("浑浊度")) {
87
+            return "ZD001";
88
+        }
89
+        
90
+        return fieldName; // 无法映射时返回原值
91
+    }
92
+}

+ 155
- 0
water-management-system/src/main/java/com/waterquality/data/UnitConverter.java Просмотреть файл

1
+package com.waterquality.data;
2
+
3
+import java.util.HashMap;
4
+import java.util.Map;
5
+
6
+/**
7
+ * 单位转换器
8
+ * 支持水利行业标准单位之间的转换
9
+ */
10
+public class UnitConverter {
11
+    
12
+    /**
13
+     * 单位转换因子表
14
+     */
15
+    private static final Map<String, Double> UNIT_CONVERSIONS = new HashMap<>();
16
+    
17
+    static {
18
+        // 流量单位转换 (m³/s 为基准)
19
+        UNIT_CONVERSIONS.put("m³/s", 1.0);
20
+        UNIT_CONVERSIONS.put("m³/min", 1.0/60.0);
21
+        UNIT_CONVERSIONS.put("m³/h", 1.0/3600.0);
22
+        UNIT_CONVERSIONS.put("L/s", 0.001);
23
+        UNIT_CONVERSIONS.put("L/min", 0.001/60.0);
24
+        UNIT_CONVERSIONS.put("L/h", 0.001/3600.0);
25
+        
26
+        // 压力单位转换 (MPa 为基准)
27
+        UNIT_CONVERSIONS.put("MPa", 1.0);
28
+        UNIT_CONVERSIONS.put("kPa", 0.001);
29
+        UNIT_CONVERSIONS.put("Pa", 0.000001);
30
+        UNIT_CONVERSIONS.put("bar", 0.1);
31
+        UNIT_CONVERSIONS.put("kgf/cm²", 0.0980665);
32
+        
33
+        // 水位单位转换 (m 为基准)
34
+        UNIT_CONVERSIONS.put("m", 1.0);
35
+        UNIT_CONVERSIONS.put("cm", 0.01);
36
+        UNIT_CONVERSIONS.put("mm", 0.001);
37
+        UNIT_CONVERSIONS.put("km", 1000.0);
38
+        
39
+        // 浊度单位转换 (NTU 为基准)
40
+        UNIT_CONVERSIONS.put("NTU", 1.0);
41
+        UNIT_CONVERSIONS.put("FTU", 1.0); // 浊度单位通常等同
42
+        UNIT_CONVERSIONS.put("mg/L", 1.0); // 简化处理,实际需要更复杂的转换
43
+    }
44
+    
45
+    /**
46
+     * 单位编码映射
47
+     */
48
+    private static final Map<String, String> UNIT_CODING = new HashMap<>();
49
+    
50
+    static {
51
+        // 标准单位编码
52
+        UNIT_CODING.put("m³/s", "FLOW001");
53
+        UNIT_CODING.put("MPa", "PRESS001");
54
+        UNIT_CODING.put("m", "LEVEL001");
55
+        UNIT_CODING.put("NTU", "TURB001");
56
+        
57
+        // 常用单位编码
58
+        UNIT_CODING.put("m³/h", "FLOW002");
59
+        UNIT_CODING.put("kPa", "PRESS002");
60
+        UNIT_CODING.put("cm", "LEVEL002");
61
+        UNIT_CODING.put("FTU", "TURB002");
62
+    }
63
+    
64
+    /**
65
+     * 单位转换
66
+     * @param value 原始值
67
+     * @param fromUnit 原始单位
68
+     * @param toUnit 目标单位
69
+     * @return 转换后的值
70
+     * @throws IllegalArgumentException 不支持的单位转换
71
+     */
72
+    public static double convert(double value, String fromUnit, String toUnit) {
73
+        if (fromUnit.equals(toUnit)) {
74
+            return value;
75
+        }
76
+        
77
+        Double fromFactor = UNIT_CONVERSIONS.get(fromUnit);
78
+        Double toFactor = UNIT_CONVERSIONS.get(toUnit);
79
+        
80
+        if (fromFactor == null || toFactor == null) {
81
+            throw new IllegalArgumentException("Unsupported unit conversion: " + fromUnit + " -> " + toUnit);
82
+        }
83
+        
84
+        return value * fromFactor / toFactor;
85
+    }
86
+    
87
+    /**
88
+     * 获取单位编码
89
+     * @param unit 单位名称
90
+     * @return 单位编码
91
+     */
92
+    public static String getUnitCode(String unit) {
93
+        return UNIT_CODING.getOrDefault(unit, "OTHER001");
94
+    }
95
+    
96
+    /**
97
+     * 获取所有支持的单位
98
+     * @return 单位列表
99
+     */
100
+    public static Map<String, String> getSupportedUnits() {
101
+        Map<String, String> units = new HashMap<>();
102
+        
103
+        // 流量单位
104
+        units.put("m³/s", "立方米/秒");
105
+        units.put("m³/min", "立方米/分钟");
106
+        units.put("m³/h", "立方米/小时");
107
+        units.put("L/s", "升/秒");
108
+        units.put("L/min", "升/分钟");
109
+        units.put("L/h", "升/小时");
110
+        
111
+        // 压力单位
112
+        units.put("MPa", "兆帕");
113
+        units.put("kPa", "千帕");
114
+        units.put("Pa", "帕");
115
+        units.put("bar", "巴");
116
+        units.put("kgf/cm²", "千克力/平方厘米");
117
+        
118
+        // 水位单位
119
+        units.put("m", "米");
120
+        units.put("cm", "厘米");
121
+        units.put("mm", "毫米");
122
+        units.put("km", "千米");
123
+        
124
+        // 浊度单位
125
+        units.put("NTU", "浊度单位");
126
+        units.put("FTU", "浊度单位");
127
+        units.put("mg/L", "毫克/升");
128
+        
129
+        return units;
130
+    }
131
+    
132
+    /**
133
+     * 获取标准单位
134
+     * @param field 字段类型
135
+     * @return 标准单位
136
+     */
137
+    public static String getStandardUnit(String field) {
138
+        switch (field.toLowerCase()) {
139
+            case "flow":
140
+            case "流量":
141
+                return "m³/s";
142
+            case "pressure":
143
+            case "压力":
144
+                return "MPa";
145
+            case "waterlevel":
146
+            case "水位":
147
+                return "m";
148
+            case "turbidity":
149
+            case "浊度":
150
+                return "NTU";
151
+            default:
152
+                return "m³/s"; // 默认流量单位
153
+        }
154
+    }
155
+}

+ 250
- 0
water-management-system/src/main/java/com/waterquality/service/DataGovernanceService.java Просмотреть файл

1
+package com.waterquality.service;
2
+
3
+import com.waterquality.data.DataNormalizer;
4
+import com.waterquality.data.StandardsMapping;
5
+import com.waterquality.data.UnitConverter;
6
+import org.springframework.stereotype.Service;
7
+
8
+import java.util.HashMap;
9
+import java.util.List;
10
+import java.util.Map;
11
+
12
+/**
13
+ * 数据治理服务
14
+ * 提供数据标准化、格式转换、质量检查等业务逻辑
15
+ */
16
+@Service
17
+public class DataGovernanceService {
18
+    
19
+    /**
20
+     * 批量数据标准化
21
+     * @param dataList 数据列表
22
+     * @return 标准化后的数据列表
23
+     */
24
+    public List<Map<String, Object>> batchNormalizeData(List<Map<String, Object>> dataList) {
25
+        return dataList.stream()
26
+                .map(DataNormalizer::normalizeData)
27
+                .toList();
28
+    }
29
+    
30
+    /**
31
+     * 批量数据质量检查
32
+     * @param dataList 数据列表
33
+     * @return 质量检查结果列表
34
+     */
35
+    public List<Map<String, Object>> batchCheckDataQuality(List<Map<String, Object>> dataList) {
36
+        return dataList.stream()
37
+                .map(DataNormalizer::checkDataQuality)
38
+                .toList();
39
+    }
40
+    
41
+    /**
42
+     * 数据转换管道
43
+     * @param data 原始数据
44
+     * @param conversionRules 转换规则
45
+     * @return 转换后的数据
46
+     */
47
+    public Map<String, Object> dataPipeline(Map<String, Object> data, Map<String, Object> conversionRules) {
48
+        Map<String, Object> result = new HashMap<>();
49
+        
50
+        // 1. 数据标准化
51
+        Map<String, Object> normalizedData = DataNormalizer.normalizeData(data);
52
+        
53
+        // 2. 应用转换规则
54
+        if (conversionRules != null) {
55
+            normalizedData = applyConversionRules(normalizedData, conversionRules);
56
+        }
57
+        
58
+        // 3. 数据质量检查
59
+        Map<String, Object> qualityCheck = DataNormalizer.checkDataQuality(normalizedData);
60
+        
61
+        result.put("normalizedData", normalizedData);
62
+        result.put("qualityCheck", qualityCheck);
63
+        result.put("conversionApplied", conversionRules != null);
64
+        
65
+        return result;
66
+    }
67
+    
68
+    /**
69
+     * 应用转换规则
70
+     * @param data 数据
71
+     * @param rules 转换规则
72
+     * @return 转换后的数据
73
+     */
74
+    private Map<String, Object> applyConversionRules(Map<String, Object> data, Map<String, Object> rules) {
75
+        Map<String, Object> convertedData = new HashMap<>(data);
76
+        
77
+        // 单位转换规则
78
+        if (rules.containsKey("unitConversions")) {
79
+            @SuppressWarnings("unchecked")
80
+            Map<String, Map<String, String>> unitConversions = (Map<String, Map<String, String>>) rules.get("unitConversions");
81
+            
82
+            for (Map.Entry<String, Map<String, String>> entry : unitConversions.entrySet()) {
83
+                String field = entry.getKey();
84
+                Map<String, String> conversion = entry.getValue();
85
+                
86
+                if (convertedData.containsKey(field) && conversion != null) {
87
+                    try {
88
+                        Object valueObj = convertedData.get(field);
89
+                        if (valueObj instanceof Number) {
90
+                            double value = ((Number) valueObj).doubleValue();
91
+                            String fromUnit = conversion.get("fromUnit");
92
+                            String toUnit = conversion.get("toUnit");
93
+                            
94
+                            if (fromUnit != null && toUnit != null) {
95
+                                double convertedValue = UnitConverter.convert(value, fromUnit, toUnit);
96
+                                convertedData.put(field, convertedValue);
97
+                            }
98
+                        }
99
+                    } catch (Exception e) {
100
+                        // 转换失败,保留原值
101
+                    }
102
+                }
103
+            }
104
+        }
105
+        
106
+        // 字段映射规则
107
+        if (rules.containsKey("fieldMappings")) {
108
+            @SuppressWarnings("unchecked")
109
+            Map<String, String> fieldMappings = (Map<String, String>) rules.get("fieldMappings");
110
+            
111
+            for (Map.Entry<String, String> entry : fieldMappings.entrySet()) {
112
+                String oldField = entry.getKey();
113
+                String newField = entry.getValue();
114
+                
115
+                if (convertedData.containsKey(oldField)) {
116
+                    Object value = convertedData.get(oldField);
117
+                    convertedData.remove(oldField);
118
+                    convertedData.put(newField, value);
119
+                }
120
+            }
121
+        }
122
+        
123
+        // 数据过滤规则
124
+        if (rules.containsKey("filterFields")) {
125
+            @SuppressWarnings("unchecked")
126
+            List<String> filterFields = (List<String>) rules.get("filterFields");
127
+            
128
+            convertedData.keySet().removeIf(key -> !filterFields.contains(key));
129
+        }
130
+        
131
+        return convertedData;
132
+    }
133
+    
134
+    /**
135
+     * 生成数据治理报告
136
+     * @param processedData 处理后的数据
137
+     * @return 治理报告
138
+     */
139
+    public Map<String, Object> generateGovernanceReport(Map<String, Object> processedData) {
140
+        Map<String, Object> report = new HashMap<>();
141
+        
142
+        // 基本信息
143
+        report.put("generatedAt", System.currentTimeMillis());
144
+        report.put("totalRecords", processedData.size());
145
+        
146
+        // 质量统计
147
+        if (processedData.containsKey("qualityCheck")) {
148
+            @SuppressWarnings("unchecked")
149
+            Map<String, Object> qualityCheck = (Map<String, Object>) processedData.get("qualityCheck");
150
+            
151
+            report.put("qualityMetrics", qualityCheck);
152
+        }
153
+        
154
+        // 标准化统计
155
+        int normalizedFields = 0;
156
+        for (String field : processedData.keySet()) {
157
+            if (field.startsWith("LL") || field.startsWith("YL") || field.startsWith("SW") || field.startsWith("ZD")) {
158
+                normalizedFields++;
159
+            }
160
+        }
161
+        report.put("normalizedFields", normalizedFields);
162
+        
163
+        // 建议
164
+        report.put("recommendations", generateRecommendations(processedData));
165
+        
166
+        return report;
167
+    }
168
+    
169
+    /**
170
+     * 生成改进建议
171
+     * @param data 数据
172
+     * @return 建议列表
173
+     */
174
+    private List<String> generateRecommendations(Map<String, Object> data) {
175
+        java.util.List<String> recommendations = new java.util.ArrayList<>();
176
+        
177
+        // 检查数据完整性
178
+        if (data.containsKey("qualityCheck")) {
179
+            @SuppressWarnings("unchecked")
180
+            Map<String, Object> qualityCheck = (Map<String, Object>) data.get("qualityCheck");
181
+            
182
+            Double completenessRate = (Double) qualityCheck.get("completenessRate");
183
+            if (completenessRate != null && completenessRate < 95.0) {
184
+                recommendations.add("数据完整性较低,建议增加必填字段的验证");
185
+            }
186
+            
187
+            Double validityRate = (Double) qualityCheck.get("validityRate");
188
+            if (validityRate != null && validityRate < 90.0) {
189
+                recommendations.add("数据有效性不足,建议加强数据格式验证");
190
+            }
191
+        }
192
+        
193
+        // 检查字段标准化
194
+        boolean hasNonStandardFields = false;
195
+        for (String field : data.keySet()) {
196
+            if (!field.startsWith("LL") && !field.startsWith("YL") && !field.startsWith("SW") && !field.startsWith("ZD")) {
197
+                hasNonStandardFields = true;
198
+                break;
199
+            }
200
+        }
201
+        
202
+        if (hasNonStandardFields) {
203
+            recommendations.add("存在非标准字段,建议应用字段映射规则");
204
+        }
205
+        
206
+        // 默认建议
207
+        if (recommendations.isEmpty()) {
208
+            recommendations.add("数据质量良好,继续保持");
209
+        }
210
+        
211
+        return recommendations;
212
+    }
213
+    
214
+    /**
215
+     * 获取治理规则模板
216
+     * @return 规则模板
217
+     */
218
+    public Map<String, Object> getRuleTemplate() {
219
+        Map<String, Object> template = new HashMap<>();
220
+        
221
+        // 单位转换规则模板
222
+        Map<String, Map<String, String>> unitConversionTemplate = new HashMap<>();
223
+        Map<String, String> flowTemplate = new HashMap<>();
224
+        flowTemplate.put("fromUnit", "m³/h");
225
+        flowTemplate.put("toUnit", "m³/s");
226
+        unitConversionTemplate.put("流量", flowTemplate);
227
+        
228
+        Map<String, String> pressureTemplate = new HashMap<>();
229
+        pressureTemplate.put("fromUnit", "kPa");
230
+        pressureTemplate.put("toUnit", "MPa");
231
+        unitConversionTemplate.put("压力", pressureTemplate);
232
+        
233
+        template.put("unitConversions", unitConversionTemplate);
234
+        
235
+        // 字段映射规则模板
236
+        Map<String, String> fieldMappingTemplate = new HashMap<>();
237
+        fieldMappingTemplate.put("自定义流量字段", "LL001");
238
+        fieldMappingTemplate.put("自定义压力字段", "YL001");
239
+        fieldMappingTemplate.put("自定义水位字段", "SW001");
240
+        fieldMappingTemplate.put("自定义浊度字段", "ZD001");
241
+        
242
+        template.put("fieldMappings", fieldMappingTemplate);
243
+        
244
+        // 过滤规则模板
245
+        java.util.List<String> filterTemplate = java.util.Arrays.asList("LL001", "YL001", "SW001", "ZD001");
246
+        template.put("filterFields", filterTemplate);
247
+        
248
+        return template;
249
+    }
250
+}

+ 196
- 0
water-management-system/src/main/java/com/waterquality/service/DataGovernanceServiceTest.java Просмотреть файл

1
+package com.waterquality.service;
2
+
3
+import com.waterquality.data.StandardsMapping;
4
+import com.waterquality.data.UnitConverter;
5
+import org.junit.jupiter.api.BeforeEach;
6
+import org.junit.jupiter.api.Test;
7
+import org.springframework.beans.factory.annotation.Autowired;
8
+import org.springframework.boot.test.context.SpringBootTest;
9
+
10
+import java.util.Arrays;
11
+import java.util.HashMap;
12
+import java.util.List;
13
+import java.util.Map;
14
+
15
+import static org.junit.jupiter.api.Assertions.*;
16
+
17
+/**
18
+ * 数据治理服务测试
19
+ */
20
+@SpringBootTest
21
+public class DataGovernanceServiceTest {
22
+    
23
+    @Autowired
24
+    private DataGovernanceService dataGovernanceService;
25
+    
26
+    private Map<String, Object> testData;
27
+    private Map<String, Object> conversionRules;
28
+    
29
+    @BeforeEach
30
+    void setUp() {
31
+        // 测试数据
32
+        testData = new HashMap<>();
33
+        testData.put("流量", "1000");
34
+        testData.put("压力", "200");
35
+        testData.put("水位", "150.5");
36
+        testData.put("浊度", "10.5");
37
+        testData.put("检测时间", "2024-01-15 10:30:00");
38
+        testData.put("备注", "系统正常运行");
39
+        
40
+        // 转换规则
41
+        conversionRules = new HashMap<>();
42
+        
43
+        // 单位转换规则
44
+        Map<String, Map<String, String>> unitConversions = new HashMap<>();
45
+        Map<String, String> flowConversion = new HashMap<>();
46
+        flowConversion.put("fromUnit", "m³/h");
47
+        flowConversion.put("toUnit", "m³/s");
48
+        unitConversions.put("流量", flowConversion);
49
+        
50
+        Map<String, String> pressureConversion = new HashMap<>();
51
+        pressureConversion.put("fromUnit", "kPa");
52
+        pressureConversion.put("toUnit", "MPa");
53
+        unitConversions.put("压力", pressureConversion);
54
+        
55
+        conversionRules.put("unitConversions", unitConversions);
56
+        
57
+        // 字段映射规则
58
+        Map<String, String> fieldMappings = new HashMap<>();
59
+        fieldMappings.put("自定义流量字段", "LL001");
60
+        fieldMappings.put("自定义压力字段", "YL001");
61
+        fieldMappings.put("自定义水位字段", "SW001");
62
+        fieldMappings.put("自定义浊度字段", "ZD001");
63
+        conversionRules.put("fieldMappings", fieldMappings);
64
+    }
65
+    
66
+    @Test
67
+    void testBatchNormalizeData() {
68
+        List<Map<String, Object>> dataList = Arrays.asList(testData);
69
+        List<Map<String, Object>> normalizedList = dataGovernanceService.batchNormalizeData(dataList);
70
+        
71
+        assertNotNull(normalizedList);
72
+        assertEquals(1, normalizedList.size());
73
+        
74
+        Map<String, Object> normalizedData = normalizedList.get(0);
75
+        assertTrue(normalizedData.containsKey("LL001"));
76
+        assertTrue(normalizedData.containsKey("YL001"));
77
+        assertTrue(normalizedData.containsKey("SW001"));
78
+        assertTrue(normalizedData.containsKey("ZD001"));
79
+    }
80
+    
81
+    @Test
82
+    void testBatchCheckDataQuality() {
83
+        List<Map<String, Object>> dataList = Arrays.asList(testData);
84
+        List<Map<String, Object>> qualityList = dataGovernanceService.batchCheckDataQuality(dataList);
85
+        
86
+        assertNotNull(qualityList);
87
+        assertEquals(1, qualityList.size());
88
+        
89
+        Map<String, Object> qualityCheck = qualityList.get(0);
90
+        assertTrue(qualityCheck.containsKey("totalFields"));
91
+        assertTrue(qualityCheck.containsKey("nullFields"));
92
+        assertTrue(qualityCheck.containsKey("invalidFields"));
93
+        assertTrue(qualityCheck.containsKey("validFields"));
94
+        assertTrue(qualityCheck.containsKey("completenessRate"));
95
+        assertTrue(qualityCheck.containsKey("validityRate"));
96
+    }
97
+    
98
+    @Test
99
+    void testDataPipeline() {
100
+        Map<String, Object> result = dataGovernanceService.dataPipeline(testData, conversionRules);
101
+        
102
+        assertNotNull(result);
103
+        assertTrue(result.containsKey("normalizedData"));
104
+        assertTrue(result.containsKey("qualityCheck"));
105
+        assertTrue(result.containsKey("conversionApplied"));
106
+        
107
+        @SuppressWarnings("unchecked")
108
+        Map<String, Object> normalizedData = (Map<String, Object>) result.get("normalizedData");
109
+        assertTrue(normalizedData.containsKey("LL001"));
110
+        assertTrue(normalizedData.containsKey("YL001"));
111
+        assertTrue(normalizedData.containsKey("SW001"));
112
+        assertTrue(normalizedData.containsKey("ZD001"));
113
+    }
114
+    
115
+    @Test
116
+    void testGenerateGovernanceReport() {
117
+        Map<String, Object> processedData = new HashMap<>();
118
+        processedData.put("LL001", 1000.0);
119
+        processedData.put("YL001", 200.0);
120
+        processedData.put("SW001", 150.5);
121
+        processedData.put("ZD001", 10.5);
122
+        
123
+        Map<String, Object> qualityCheck = new HashMap<>();
124
+        qualityCheck.put("totalFields", 4);
125
+        qualityCheck.put("nullFields", 0);
126
+        qualityCheck.put("invalidFields", 0);
127
+        qualityCheck.put("validFields", 4);
128
+        qualityCheck.put("completenessRate", 100.0);
129
+        qualityCheck.put("validityRate", 100.0);
130
+        
131
+        processedData.put("qualityCheck", qualityCheck);
132
+        
133
+        Map<String, Object> report = dataGovernanceService.generateGovernanceReport(processedData);
134
+        
135
+        assertNotNull(report);
136
+        assertTrue(report.containsKey("generatedAt"));
137
+        assertTrue(report.containsKey("totalRecords"));
138
+        assertTrue(report.containsKey("normalizedFields"));
139
+        assertTrue(report.containsKey("recommendations"));
140
+        assertTrue(report.containsKey("qualityMetrics"));
141
+    }
142
+    
143
+    @Test
144
+    void testGetRuleTemplate() {
145
+        Map<String, Object> template = dataGovernanceService.getRuleTemplate();
146
+        
147
+        assertNotNull(template);
148
+        assertTrue(template.containsKey("unitConversions"));
149
+        assertTrue(template.containsKey("fieldMappings"));
150
+        assertTrue(template.containsKey("filterFields"));
151
+        
152
+        @SuppressWarnings("unchecked")
153
+        Map<String, Map<String, String>> unitConversions = (Map<String, Map<String, String>>) template.get("unitConversions");
154
+        assertTrue(unitConversions.containsKey("流量"));
155
+        assertTrue(unitConversions.containsKey("压力"));
156
+        
157
+        @SuppressWarnings("unchecked")
158
+        Map<String, String> fieldMappings = (Map<String, String>) template.get("fieldMappings");
159
+        assertTrue(fieldMappings.containsKey("自定义流量字段"));
160
+        assertTrue(fieldMappings.containsKey("自定义压力字段"));
161
+        
162
+        @SuppressWarnings("unchecked")
163
+        List<String> filterFields = (List<String>) template.get("filterFields");
164
+        assertTrue(filterFields.contains("LL001"));
165
+        assertTrue(filterFields.contains("YL001"));
166
+        assertTrue(filterFields.contains("SW001"));
167
+        assertTrue(filterFields.contains("ZD001"));
168
+    }
169
+    
170
+    @Test
171
+    void testUnitConversion() {
172
+        double value = 1000;
173
+        String fromUnit = "m³/h";
174
+        String toUnit = "m³/s";
175
+        
176
+        double convertedValue = UnitConverter.convert(value, fromUnit, toUnit);
177
+        assertEquals(1000.0 / 3600.0, convertedValue, 0.001);
178
+        
179
+        String unitCode = UnitConverter.getUnitCode(toUnit);
180
+        assertEquals("FLOW001", unitCode);
181
+    }
182
+    
183
+    @Test
184
+    void testStandardsMapping() {
185
+        Map<String, Object> flowField = StandardsMapping.getStandardField("flow");
186
+        assertNotNull(flowField);
187
+        assertEquals("LL", flowField.get("standard"));
188
+        assertEquals("m³/s", flowField.get("unit"));
189
+        
190
+        String standardField = StandardsMapping.mapToStandardField("自定义流量字段");
191
+        assertEquals("LL001", standardField);
192
+        
193
+        String standardField2 = StandardsMapping.mapToStandardField("flow");
194
+        assertEquals("LL001", standardField2);
195
+    }
196
+}

+ 50
- 0
water-management-system/src/main/resources/application.yml Просмотреть файл

1
+server:
2
+  port: 8080
3
+  servlet:
4
+    context-path: /
5
+
6
+spring:
7
+  application:
8
+    name: water-management-system
9
+  
10
+  datasource:
11
+    url: jdbc:h2:mem:testdb
12
+    driver-class-name: org.h2.Driver
13
+    username: sa
14
+    password: password
15
+    hikari:
16
+      maximum-pool-size: 10
17
+      minimum-idle: 2
18
+  
19
+  h2:
20
+    console:
21
+      enabled: true
22
+      path: /h2-console
23
+  
24
+  jpa:
25
+    hibernate:
26
+      ddl-auto: create-drop
27
+    show-sql: false
28
+    properties:
29
+      hibernate:
30
+        format_sql: true
31
+        dialect: org.hibernate.dialect.H2Dialect
32
+  
33
+  mvc:
34
+    pathmatch:
35
+      matching-strategy: ant_path_matcher
36
+
37
+logging:
38
+  level:
39
+    com.waterquality: DEBUG
40
+    org.springframework.web: INFO
41
+    org.hibernate: WARN
42
+
43
+management:
44
+  endpoints:
45
+    web:
46
+      exposure:
47
+        include: health,info
48
+  endpoint:
49
+    health:
50
+      show-details: always