TV Weathercaster Evaluation: An Atmospheric Sciences 101 Class Project
For nine weeks this autumn, over 100 students in the UW Atmospheric Sciences 101 class taught by Professor Clifford Mass watched TV weather segments on late-evening local news broadcasts in order to evaluate the relative skill of TV weathercasters and the National Weather Service for maxiimum temperature forecasts at Seattle-Tacoma Airport.
Major questions included:
How rapidly does forecast skill deteriorate as a function of forecast projection? That is, how much worse are 4-5 day forecasts compared to 1-2 day forecasts?
Does skill vary among local TV stations?
Are some TV stations consistently more skillful in forecasting maximum temperature?
Do any significantly improve upon the NWS forecasts?
Precipitation forecasts were not verified since TV stations in the Seattle market do not provide any quantifiable prediction s of precipitation (using graphics to show collections of drops, etc when rain is forecast).
Method:
Two to three students were assigned to each broadcast to reduce the possibility of transcribing bad data. The late night (10 PM or 11 PM) broadcasts were used. For the NWS forecasts for the first two days into the forecast period we used the official forecast for Seattle that is released at approximately 9:30 PM. Since there is no official NWS forecast for days 3-5 we used the forecasts provided by the National Weather Service model output statistics package for the Medium Range Forecast model. All of the above data was collected daily and the collection was entered into computer files by Scott Judson, an undergraduate. The verification site for the maximum temperature forecast was Seattle Tacoma Airport--the site generally used for verification purposes in the central sound. Mr. Judson and Ernie Recker, a department staff member, wrote a simple computer program to analyze the information--which is presented below. Below are the raw results; conclusion are presented at the end. It is important to note that small differences in forecast skill (less than approximately .5F) are probably not statistically significant.
Day 1: Mean Maximum Temperature Errors
National Weather Service Mean Error: 2.5F
|
Error (F) |
Stations |
|
2.3 to 2.7 |
NWS, KOMO, KING, KIRO |
|
2.8 to 3.2 |
KSTW |
Day 2: Mean Maximum Temperature Errors
National Weather Service Mean Error: 2.8F
|
Error (F) |
Stations |
|
2.8 to 3.2 |
NWS, KOMO, KING, KIRO, KSTW |
|
3.3 and higher |
none |
Day 3: Mean Maximum Temperature Errors
National Weather Service Mean Error: 3.3F
|
Error (F) |
Stations |
|
3.3 to 3.7 |
NWS, KOMO, KING, KIRO, KSTW |
|
3.8 and higher |
none |
Day 4: Mean Maximum Temperature Errors
National Weather Service Mean Error: 3.2F
|
Error (F) |
Stations |
|
3.2 |
NWS, KOMO |
|
3.3 to 3.7 |
KIRO |
|
3.8 to 4.2 |
KING, KSTW |
Day 5: Mean Maximum Temperature Errors
National Weather Service Mean Error: 4.9F
|
Error (F) |
Stations |
|
3.8 to 4.2 |
KIRO |
|
4.3 to 4.7 |
KING |
|
4.8 to 5.2 |
NWS, KSTW |
Conclusions:
In general, for the first three days into the forecast, local television stations do not deviate greatly from National Weather Service maximum temperature forecasts and thus differences in skill between them are relatively small. For the next day forecast (Day 1), the all the TV stations except KSTW had a mean error within .2F of the NWS error (2.5F) . On Day 2, the mean errors at all of the TV stations were within .4F of the NWS error of 2.8F. At Day 3, the forecasts were even more uniform, with the mean errors at all stations being with .2F of the NWS average error of 3.3F. Variations in forecast skill increase considerably on days 4 and 5.
Another interesting finding is that forecast error increases slowly during the first few days but then increases rapidly on the fifth day, reaching levels nearly twice that noted on day 1.