Analysis of the Optimal Window Size of Hampel Filter for Calibration of Real-time Water Level in Agricultural Reservoirs

Dong-Hyuk Joo; Ra Na; Ha-Young Kim; Gyu-Hoon Choi; Jae-Hwan Kwon; Seung-Hwan Yoo

doi:10.5389/KSAE.2022.64.3.009

All Issue

2022 Vol.64, Issue 3 Preview Page Next Page

Analysis of the Optimal Window Size of Hampel Filter for Calibration of Real-time Water Level in Agricultural Reservoirs 농업용저수지의 실시간 수위 보정을 위한 Hampel Filter의 최적 Window Size 분석

2022. pp. 9-24

PDF XML

Abstract

Currently, a vast amount of hydrologic data is accumulated in real-time through automatic water level measuring instruments in agricultural reservoirs. At the same time, false and missing data points are also increasing. The applicability and reliability of quality control of hydrological data must be secured for efficient agricultural water management through calculation of water supply and disaster management. Considering the characteristics of irregularities in hydrological data caused by irrigation water usage and rainfall pattern, the Korea Rural Community Corporation is currently applying the Hampel filter as a water level data quality management method. This method uses window size as a key parameter, and if window size is large, distortion of data may occur and if window size is small, many outliers are not removed which reduces the reliability of the corrected data. Thus, selection of the optimal window size for individual reservoir is required. To ensure reliability, we compared and analyzed the RMSE (Root Mean Square Error) and NSE (Nash–Sutcliffe model efficiency coefficient) of the corrected data and the daily water level of the RIMS (Rural Infrastructure Management System) data, and the automatic outlier detection standards used by the Ministry of Environment. To select the optimal window size, we used the classification performance evaluation index of the error matrix and the rainfall data of the irrigation period, showing the optimal values at 3 h. The efficient reservoir automatic calibration technique can reduce manpower and time required for manual calibration, and is expected to improve the reliability of water level data and the value of water resources.

Keywords

Hampel filter

agricultural reservoir

window size

reliability

References

REFERENCES

J. H. Bang, Y. H. Lee, S. Y. Jeong and J. Y. Choi, Environmental Protection Agency, Athens, GA, A study of outlier detection on time series of water-level in agricultural reservoir. In Proceedings of the Korean Society of Agricultural Engineers Coneference 60, (in Korean) (2017)

J. Y. Choi, Korea Rural Community Corporation, Development of quality control methods for water level data and irrigation water supply estimation, 4-10, (in Korean) (2018)

G. H. Choi, K. H. Kim, S. Y. Lim, S. H. Yoo, E. H. Choi and M. S. Kang, In Proceedings of the Korean Society of Agricultural Engineers Conference 107, Quality control on water-level in irrigation canal using filter techniques, (in Korean) (2020)

Rob Hyndman, J. Koehler and B Anne, International Journal of Forecasting, “Another look at measures of forecast accuracy”, 22(4); 679-688 (2006)10.1016/j.ijforecast.2006.03.001

H. K. Kim and S. W. Park, Korean Journal of Hydrosciences, Development of hydrologic data management system based on relational database, 39(10); 855-866 (2006)10.3741/jkwra.2006.39.10.855

H. S. Kim and C. S. Kim, Korean Society of Civil Engineers Conference, Hydrological data quality control system for monitoring and improving the data quality; 2032-2036, (in Korean) (2003)

M. G. Kim, J. Y. Choi, J. H. Bang and J. J. Lee, Journal of the Korean Society of Agricultural Engineers, Outlier detection of real-time reservoir water level data using threshold model and artificial neural network model, 61(1); 107-120, (in Korean) (2019)10.5389/KSAE.2019.61.1. 107

Korea Rural Community Corporation (KRC), Development of technology for securing reliability of water level measurement data and estimating water supply, (in Korean) (2018)

Korea Rural Community Corporation (KRC), A Study on the establishment of quality control standards for hydrologic data in agricultural reservoirs and waterways, (in Korean) (2019)

Korea Rural Community Corporation (KRC), Advanced quality management study of reservoir water level measurement data, (in Korean) (2020)

K-water, K-water Data Quality Management Guidelines (Water Information), (in Korean) (2018)

J. W Lee, H. S. Kim, C. H. Hwang, I. S. Kang and H. K. Jeong, In Proceedings of the Korean Institute of Information and Communication Sciences Conference, Data cleansing algorithm for reducing outlier; 342-344, (in Korean) (2018)

MathWorks, https://kr.mathworks.com

Ministry of Agriculture, Food and Rural Affairs, Development of information analysis technology for abnormal behavior of agricultural reservoirs using big data related to weather information and water levels, (in Korean) (2016)

Ministry of Environment, Environment and maintenance and management of hydrologic research facilities and standards for quality management of hydrologic data, (in Korean) (2018)

Ministry of Land, Infrastructure and Transport, Establishment and operation of the national hydrologic data quality management system (7th), (in Korean) (2018)

Ministry of Land, Transport and Maritime Affairs, Establishment of a basic plan for hydrologic investigation(2010-2019), (in Korean) (2008)

Ministry of Land, Transport and Maritime Affairs, The 4th Comprehensive Plan for Water Resources 2nd Amendment (2011-2020), (in Korean) (2011)

Ministry of the Interior and Safety, Public Data Quality Management Manual ver 2.0, (in Korean) (2018)

J. E. Nash and J. V Sutcliffe, Journal of Hydrology, “River flow forecasting through conceptual models part I - A discussion of principles”, 10(3); 282-290 (1970)10.1016/0022-1694(70)90255-6

C. R. Oh, S. C. Park, H. M. Lee and Y. P. Pyo, Journal of The Korean Society of Civil Engineers, A forecasting of water quality in the Youngsan river using neural network, 22(3B); 372-382, (in Korean) (2002)

R. K. Pearson, Y. Neuvo, J. Astola and M Gabbouj, EURASIP Journal on Advances in Signal Processing, Generalized Hampel filters, 2016(1); 1-18 (2016)10.1186/s13634-016-0383-6

W. H. Press, B. P. Flannery, S. A. Teukolsky and W. T Vetterling, Numerical recipes : the art of scientific computing, Cambridge University Press (1987)10.1063/1.2820230

Pukelsheim, Friedrich, The American Statistician, “The three sigma rule.”, 48(2); 88-91 (1994)10.1063/1.2820230

K. G. Ranjan, D. S. Tripathy, B. R. Prusty and D Jena, International Journal of Numerical Modelling: Electronic Networks, Devices and Fields, An improved sliding window prediction-based outlier detection and correction for volatile time-series, 34(1); e2816 (2021)10.1002/jnm.2816

H. J. Shin and T. H. Lee, In Proceedings of the Korea Water Resources Association Conference, A study on the estimation of missing hydrological data using adaptive network-based fuzzy inference system (anfis); 264, (in Korean) (2020)

United States Geological Survey, South Carolina:United States Geological Survey, Computing discharge using the index velocity method; 35-59 (2012)

M. H Yang, W. H. Nam, H. J. Kim, T. G. Kim, A. K. Shin and M. S. Kang, Journal of the Korean Society of Hazard Mitigation, Anomaly detection in reservoir water level data using the LSTM model based on deep learning, 21(1); 71-81, (in Korean) (2021)10.9798/kosham.2021.21.1.71

Information

Publisher :The Korean Society of Agricultural Engineers
Publisher(Ko) :한국농공학회
Journal Title :Journal of Korean Society of Agricultural Engineers
Journal Title(Ko) :한국농공학회논문집
Volume : 64
No :3
Pages :9-24
Received Date : 2021-12-16
Revised Date : 2022-03-03
Accepted Date : 2022-03-28
DOI :https://doi.org/10.5389/KSAE.2022.64.3.009

Journal of Korean Society of Agricultural Engineers ISSN:1738-3692(Print) 2093-7709(Online) 한국농공학회논문집

All Issue

REFERENCES