QUALITY ASSURANCE

Proper Quality Assurance and Quality Control (QA/QC) protocols are essential to WOW. We have gone to great lengths to assure the accuracy of our data - the following sections describe these measures in detail.

What are Quality Assurance and Quality Control?

QA/QC basically refers to all those things good investigators do to make sure their measurements are right on (accurate; the absolute true value), reproducible (precise; consistent), and have a good estimate of their uncertainty. In the regulatory arena, this aspect of data collection is as crucial to the final outcome of a confrontation as the numbers themselves. It specifically involves following established rules in the field and lab to assure everyone that the sample is representative of the site, free from outside contamination by the sample collector (no dirty hands touching the water) and that it has been analyzed following standard QA/QC methods. This typically involves comparing the sample to a set of known samples for estimating accuracy and by replicating the measurement to estimate its precision. The U.S. Environmental Protection Agency has lots to add should you wish to learn more of the technical aspects of a Quality Assurance Program (QAP). See also the WOW Lessons about data quality and interpretation for further information.

http://www.epa.gov/owowwtr1/monitoring/volunteer/qappexec.htm contains The Volunteer Monitor’s Guide to: Quality Assurance Project Plans. 1996. EPA 841-B-96-003, Sep 1996, U.S. EPA, Office of Wetlands, Washington, D.C. 20460, USA

DATA TYPES
There are basically two sets of environmental data that are collected for WOW:

(1) conventional water quality parameters such as nutrients (N- and P-series of nutrients), chlorophyll, clarity, fecal coliform bacteria, manual field profiles for temp, DO, EC, etc. These are based upon traditional methods where a trained staff person records measurements at different depths from a sensor lowered over the side of a boat and collects water from discrete depths that are returned to the lab for analysis.

(2) remotely sensed and controlled R.U.S.S. (Remote Underwater Sampling System) units that control the depth and sampling interval of water quality sondes housing depth, temperature, DO, pH, EC and turbidity probes. Data may be transmitted via cellular phone/modem to our base computer/website immediately upon completion of a depth profile, or may be stored on board the RUSS and downloaded less frequently (each morning, currently) to save connection costs.

Conventional data quality assurance procedures follow guidelines set by the U.S.EPA (1987; 1989a,b), and APHA (1998). Water chemistry and manual field profiles are collected by trained staff limnologists and technicians at both Hennepin Parks (HP under Principal Investigator/Limnologist John Barten's supervision) and the Natural Resources Research Institute (NRRI under Co-Principal Investigator / Limnologist Rich Axler). Both the Hennepin Parks Water Quality Laboratory and the NRRI Central Analytical Laboratory are certified annually by the Minnesota Department of Health for Federal Safe Drinking Water Act and Clean Water Act parameters (Ameel et al. 1993, 1998; Axler and Owen 1994; Archer and Barten 1995, 1996; Barten 1997; MCWD 1997). The certification procedure involves blind analyses of certified performance standards and an in-depth site inspection and interview approximately every other year. The NRRI lab has also been certified over the past decade by the Minnesota Pollution Control Agency and the Minnesota Department of Natural Resources for low-level water quality analyses in pristine, acid-sensitive lake monitoring programs and for sediment contaminant analyses in the St. Louis River and Upper Mississippi Rivers.

RUSS QA/QC is performed at a number of levels. The sensors are either Hydrolab H20 or YSI 6820 probe/sonde instruments; both HP and NRRI staff follow the Instrument Manuals for calibration and maintenance procedures. Our staff also have extensive experience with these calibration procedures and with their importance in interpreting field data and distinguishing systematic errors associated with deteriorating, or bio-fouled probes. In 1998 and 1999 we gained considerable experience in dealing with problems associated with continuous sensor deployment; the resultant protocols are included in our WOW efforts. Other aspects of the data management process are discussed in Host et al. (2000a, 2000b).

NRRI contributed to the initial development of the RUSS technology. During the preliminary and early stages of Water on the Web, numerous tests were conducted in regard to the accuracy and precision of in-situ data. Since both the YSI and Hydrolab systems are well established and used for numerous state and federal monitoring programs, the principal concerns related to the time allowed for sensor equilibration at each depth . Of all the sensors that we use, dissolved oxygen is most susceptible to erroneous values from inadequate stabilization- the error being greatest in regions with steep depth-gradients in DO. Following our collaborative work on this topic with Apprise Technologies, Inc., the company subsequently ran a nearly yearlong experiment in Lake Waco, Texas with Hydrolab, Inc. comparing RUSS-transmitted data to conventional datalogger data. The data sets agreed within sensor specifications. Both sensor companies have internal quality control systems (YSI is ISO14001 registered) that guarantee the consistent quality of their sensors. Apprise has worked independently with both companies to integrate these sensor packages with their RUSS units. As a part of these programs, the RUSS technology was independently field-tested by both companies and both YSI and Hydrolab have audited the Apprise facilities for QA/QC compliance. Apprise has also implemented an internal quality system based on the ISO9000 system and has been extremely helpful in dealing with problems that occasionally arise with the Lake Access and WOW units. A more complete description of our current protocols follows:

RUSS SENSOR RESOLUTION & REPORTING LIMITS

On the RUSS unit, the on-board computer processes a user-submitted instruction sequence, the sensor package is sent to a specified depth, and a series of feedback corrections are made until the sensors are stabilized within 0.2 m of the specified depth. Output from the sensors is monitored to assess when the readings on all parameters have stabilized to a specified criterion, usually a coefficient of variation <20% for a running set of 10 consecutive measurements over an interval of ~1 minute. Dissolved oxygen typically requires the most amount of time to stabilize on average, in part because of the occurrence of steeper depth gradients for this parameter. Depending on the site characteristics and the specific O2-sensor, as much as 3-5 minutes may be required for complete equilibration. Once stabilized, readings on all parameters are stored in buffer memory on the on-board computer. The raw data stream is a simple string of comma-delimited ASCII text containing a time signature, depth, and parameter values (Table 1).

Table 1. Output from Lake Access RUSS unit on Halsteds Bay, Lake Minnetonka, MN, 6/4/2000.

Unit: EMPT2 site: Halsteds Bay
Site	Sample	Sample	Depth	Temp	pH	EC @ 25 C	O2	O2	Turb
	Date	Time	(m)	oC		(uS/cm)	(mg/L)	(% sat)	(NTU)
Halsteds	06/04/2000	00:10:58	1	18.3	8.4	406	10.0	107	11
Halsteds	06/04/2000	00:11:43	2	18.3	8.4	407	10.1	107	6
Halsteds	06/04/2000	00:13:34	3	18.2	8.4	407	10.0	106	3
Halsteds	06/04/2000	00:15:13	4	17.9	8.3	410	9.1	97	15
Halsteds	06/04/2000	00:17:04	5	17.6	8.2	411	8.0	84	5
Halsteds	06/04/2000	00:18:55	6	17.3	8.0	414	6.7	70	4
Halsteds	06/04/2000	00:20:34	7	16.7	7.8	419	4.9	50	9
Halsteds	06/04/2000	00:22:25	8	16.3	7.6	425	1.8	18	14

To date we have set the reporting limits for RUSS data based on instrument specifications and prior knowledge of the magnitude of typical field variations. This information is presented within the RUSS data section of the WOW web site. The resolution, i.e. the smallest reading shown for a particular parameter is likely to be considerably lower than the error associated with differences in time, with depth fluctuations, and with sensor drift and calibration accuracy. Periodic examination of the RUSS data stream with Apprise Technologies, Inc. has generally confirmed the estimated accuracy reported below (Table 2). An important, and greatly underestimated element of the WOW projects has been to assess the accuracy of these data by comparison with approximately biweekly manual profiles. However, it is likely that the relative precision of the data between depths within a water column profile and within a few hours to a day will be better than from week-to-week.

Ideally, if all of the RUSS sensors behaved according to sensor-manufacturer's specifications (Table 2) we could simply post the data on the Lake Access web site and assume it is accurate to these levels. However, except for temperature, all of the sensors require routine maintenance and calibration. When using these sensors for manual profiling, that is, visiting lake sites by boat, we always re-calibrate the pH, EC and turbidity sensors using individual standard solutions with known values, and the DO by air calibration. Experience has taught us that the sensors remain stable during the course of a sampling day.

Table 2. Reporting limits for RUSS sensor data (Hydrolab or YSI sensors)

Depth (m)	Temp (oC)	DO (mg/L)	DO % saturation	pH	EC (uS/cm)	Turbidity (NTUs)
Resolution (what is reported by the RUSS sensors)
± 0.12	± 0.1	± 0.1	± 0.1	± 0.1	± 1	± 1
Estimated Accuracy (what we really trust)
± 0.3	± 0.15	± 0.2	± 2	± 0.2	± 10	± ~3

However, when deployed for continuous operation, as for the RUSS unit, the sensors are colonized gradually by a biofilm of algae and less noticeably by bacteria and fungi as well. As this material builds up, its metabolic activity interferes with the sensor's ability to accurately sample the surrounding water. One can easily picture the effect of fine filaments of algae wafting intermittently between the electrodes of the EC sensor or in the light path of the turbidimeter giving seemingly erratic values with wide swings as the sensors move up and down. An anomalous spike in the Ice Lake EC data during July 1998 (see shaded region in the Surface Trends for Ice Lake on the WOW site), are a good example of this effect and are the basis of a lab lesson (Increased Conductivity: Are Culverts The Culprits? in draft). DO and turbidity probes are most susceptible to these changes, followed by pH and EC.

SENSOR MAINTENANCE AND CALIBRATION

Lake Access and WOW staff set up the following protocols to minimize these biofouling and instrument drift effects to quality assure the RUSS data:

* Clean and re-calibrate sensors frequently (about every 2 weeks) and perform manual profiles with an independent instrument at the same time

* Compare independent manual profiles with near-simultaneous RUSS data prior to cleaning (re-calibration). This provides assurance that data from the previous period are accurate. We calculate test statistics for each parameter as:

and

for each parameter. They PASS according to rules in Table 3

Table 3. Quality Assurance Criteria for RUSS Sensors
SENSOR	RPD	DELTA
Temperature	< 5%	< 0.2 oC
DO	< 10 %	< 0.5 mg O2/L
EC	< 10 %	< 5 uS/cm
pH	< 10 %	< 0.2 units
turbidity	< 10 %	< 5 NTUS

If the data "passes", it is considered acceptable for the previous period. If not, we examine it in the context of our understanding of the limnology of the individual lake and other data (nutrients, chlorophyll, trends, etc.) and then either delete it from the database or allow it to be posted. We have to be careful not to delete anomalous data that may simply reveal real dynamic changes. The sheer volume of data (218,720,430,742,644,316,434,172,687,130 values to date) has been taxing and we lack the resources to always be as current as we would like. In the interim, data are posted as provisional . Dates of calibrations and these manual data are posted in the DATA section of WOW and are available within easily accessible Excel files - these will soon be posted on the Lake Access site as well. The three Data Visualization Tools (DVTs) developed for Lake Access and Water on the Web are also helpful in rapidly displaying the data in a variety of formats to help identify anomalous data. We are currently in the process of adding 'calibration date flags' to the control panels of the Profile Plotter and Color Mapper DVTs and to the DxT Profiler to allow the user to more easily keep track of calibration dates as the data stream is being viewed.

The first year of WOW, 1998, taught us that we were understaffed for the frequency of maintenance required for continuous RUSS operation at Ice Lake and Lake Independence. With an additional three units being deployed for 1999 and 2000, we set up collaborations with Itasca Soil and Water Conservation District (for Ice Lake), Hennepin Parks (for Lake Independence and Lake Minnetonka), the Minnesota Department of Natural Resources Regional Fisheries for Grindstone Lake, and the Minnesota Pollution Control Agency staff for the St Louis River site (still in development as of July 2000). Lake Access and WOW staff work with these folks to clean and re-calibrate all sensors approximately every 1-3 weeks depending on the site. The less productive sites (Grindstone and Ice lakes) generally require less maintenance.

DATA TRANSMISSION AND INITIAL QA SCREENING

The program that imports the RUSS data currently is scheduled to run every day at 7:30 AM. The RUSS base station software is used to call each RUSS and download data that has been collected since the last call. A file containing real-time data (RTD) collected during the duration of the call is also created. These new profile data and RTD files are stored on the base station computer as plain ASCII text files, one file for each day's data. The data files from each site are stored in a separate directory on the computer. Table 1 (above) is an sample of an original profile data file created by the RUSS base station.

The Conversion Process

A program (the importer) is now launched. It reads data files that have been created or changed since the last time it was run, and converts the data to the format used by the report generating and data visualization programs. Additionally, the original data files are copied to the web server so they are accessible for immediate QA/QC. Profile data files are copied to http://wow.nrri.umn.edu/data/ and RTD files are copied to http://wow.nrri.umn.edu/rtd/ .

The importer parses the first line of a new or modified RUSS data file and tests to make sure that the Unit and Site correspond to what is expected. If not, an error message is generated and no further action is taken with this file. This will catch errors that could occur if, for example, a data file from Halsteds Bay was somehow stored in the Lake Independence directory. Next, it reads the line containing the column descriptions, and compares it with what is expected. If it differs, an error message is generated and no further action is taken with this file. This will catch errors that could occur if, for example, a new parameter is being read by the RUSS, but the importer hasn't been updated to handle the change. Now, each data line is read and converted to a "Reading". A set of readings is combined to form a "Profile" in the data base. Specific data is rejected by the importing program if it is outside these ranges:

temperature	< -1 or > 35 oC
pH	< 5 or > 10
specific conductance (EC25)	< 1 or > 600 uS/cm
dissolved oxygen (DO)	< -1 or > 20 mg/L O2/L
DO % saturation	< -5 or > 200 %
turbidity	< -5 or > 100NTU (note: turbidity values between -5 and 0 are set = 0)

There is no direct indication in the raw data files of where one profile ends and the next begins, so the importer applies some heuristics to decide how to assign readings to profiles. The values listed below are those in current use, but they can be changed. Since only the actual time is reported on each data line, the importer assigns a "scheduled" time to the new profile, using the nearest :00 or :30 minute time value before the time reported for the first reading in the profile. Subsequent readings are added to the same profile provided that:
     1) the reading is from a lower depth, and
     2) the reading was taken within 30 minutes of the previous one

When the importer either comes to a line where the reading no longer qualifies for the current profile or it reaches the end of the data file, it will add the new profile to the data base provided that:
     1) the first reading starts within 3 meters of the surface,
     2) there are at least 4 readings in the profile, and
     3) the date is not in the future

Instead it will generate an appropriate error message in the log file, and disregard the profile. This helps eliminate partial or invalid profiles that could be caused by RUSS hardware problems.

If it is winter and the RUSS is installed on ice, we set the minimum upper depth to 1 or 2 meters to minimize the risk of the unit becoming trapped in the hole through the ice. The data importer then creates a default reading at 0 meters, listing a temperature of 0 oC, with all other parameters blank (since we don't know what their true values are). The time for this reading is set equal to the scheduled time. The timestamp of each reading is expected to be unique, and can be used as the key value in a database. There is the possibility that the first actual reading in the profile could have the same timestamp as the bogus reading, so the readings in the profile are checked for duplicate times. If found, 5 seconds are added to the time of the deeper reading, and the change is noted in the log file.

Sometimes a sensor for a particular parameter at a particular site will go bad. In this case, the importer program can be customized to reject that parameter when importing the data from the site. Data stored in buffer memory is transmitted to the base station via cellular phone at specified intervals, or at the request of the user. Standard parity-based error correction techniques are used to ensure that data were not altered during transmission. At the base station, a JAVA based application adds the raw data to a standardized relational database (DBMS) file. For archival purposes the original ASCII data are stored in a compressed data format (ZIP) file. The ASCII and DBMS files are periodically downloaded to an off-site location via File Transfer Protocol (FTP).

FINAL DATA REVIEW & POSTING

At present (July 2000), funding limitations have precluded adherence to a rigorous schedule for removing the provisional label from RUSS data. In part this is a due to the need to review ancillary water chemistry data before making final decisions when the RUSS data are questionable. All water chemistry data posted on the Lake Access and WOW sites however, have passed QA/QC prior to being posted, although this typically takes from 30-60 days after collection.

Despite regular maintenance and calibration schedules, occasional RUSS data anomalies still occur. To date, they have virtually always been associated with DO and/or turbidity data although there have been recurring problems with the pH probe at the WOW Grindstone Lake site.

The most troublesome anomalies are those that occur within the calibration window of time, are not flagged by our automated screening tools and are not unreasonable values in terms of the range of values previously measured for that depth stratum and time of year. These errors have not been trivial to identify and require careful examination in a complete limnological (lake/watershed/climate) context by a professional limnologist. The process is adequately described as Best Professional Judgement (BPJ). In some cases we have decided to adjust data by calculating correction factors when there is accurate calibration data spanning the period in question and when the results estimated by interpolation are consistent with the rest of the data set. In other cases we have simply rejected the data - omitting it from the website. Data deletions are summarized and circulated to all limnological staff and archived in a hidden section of the Lake Access and WOW websites. The WOW project sends a periodic e-mail newsletter providing data updates to all teachers and researchers using the site for educational or research purposes; you can subscribe to this newsletter at http://wow.nrri.umn.edu/wow/contactus.html.

SUMMARY

The QA/QC of near-real time remotely collected sensor data has provided challenges that were not present under traditional sampling regimes. We have attempted to develop rigorous protocols for each step of the data aquisition effort, and believe these protocols suit the needs of projects such as Lake Access and Water on the Web. Nonetheless, as these technologies become more common in resource management, future efforts must be directed toward the unique problems posed by real-time data collection.

ACKNOWLEGEMENTS

RIchard Axler, Elaine Ruzycki, and Norm Will contributed to the development, testing, and documentation of these QA/QC protocols.


REFERENCES

Ameel, J.J., Axler, R.P. and Owen, C.J. 1993. Persulfate digestion for determination of total nitrogen and phosphorus in low-nutrient waters. Amer. Environ. Labor. October 1993, p.1-11.

Ameel, J., E. Ruzycki and R.P. Axler. 1998. Analytical chemistry and quality assurance procedures for natural water samples. 6th edition. Central Analytical Laboratory, NRRI Tech. Rep. NRRI/TR?98/03.

APHA. 1998. Standard methods for the examination of water and wastewater. American Public Health Association, Washington, D.C.

Archer, A. and J. Barten. 1995. Quality assurance manual. Hennepin Parks Water Quality Laboratory. September 1995. Hennepin Parks, 3800 County Road 4, Maple Plain, MN 55359.

Archer, A. and J. Barten. 1996. Laboratory Procedures Manual. Hennepin Parks Water Quality Laboratory. October 1996. Hennepin Parks, 3800 County Road 4, Maple Plain, MN 55359.

Axler, R.P. and C.J. Owen.1994. Fluorometric measurement of chlorophyll and phaeophytin: Whom should you believe? Lake and Reservoir Management 8:143-151.

Barten, J. 1997. Water quality monitoring plan. Hennepin Parks, 3800 County Road 4, Maple Plain, MN 55359.

EPA. 1987. Handbook of methods for acid deposition studies-Laboratory analysis for water chemistry. EPA/600/4-87-026

EPA 1989a. Preparing perfect project plans. US EPA Risk Reduction Engineering Laboratory, Cincinnati, OH, EPA/600/9-89/087.

EPA.1989b. Handbook of methods for acid deposition studies-Field operations for surface water chemistry. EPA/600/4-89-020.

EPA. 1996. The Volunteer Monitor's Guide to: Quality Assurance Project Plans. EPA 841-B-96-003, Sep 1996, U.S. EPA, Office of Wetlands, Washington, D.C. 20460, USA (http://www.epa.gov/owowwtr1/monitoring/volunteer/qappexec.htm)

Host, G., N. Will, R.Axler, C. Owen and B. Munson. 2000a. Interactive technologies for collecting and visualizing water quality data. URISA Journal (In Press; refereed: http:// wow.nrri.umn.edu/urisa)

Host, G.E. , B. H. Munson, R. P. Axler, C. A. Hagley, G. Merrick and C. J. Owen. 2000b. Water on the Web: Students monitoring Minnesota rivers and lakes over the Internet. AWRA Spec.Ed. (Dec., 1999). (refereed: www.awra.org/proceedings/www99/w74/index.htm.).

MCWD. 1997. Quality assurance - quality control assessment report. Lake Minnetonka Monitoring Program 1997. Minnehaha Creek Watershed District, 2500 Shadywood Road, Excelsior, MN 55331-9578.