{"id":563532,"date":"2019-03-07T13:30:01","date_gmt":"2019-03-07T21:30:01","guid":{"rendered":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/?p=563532"},"modified":"2019-06-18T09:58:35","modified_gmt":"2019-06-18T16:58:35","slug":"predicting-the-holy-grail-of-climate-forecasting-a-new-model-and-a-new-public-dataset","status":"publish","type":"post","link":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/blog\/predicting-the-holy-grail-of-climate-forecasting-a-new-model-and-a-new-public-dataset\/","title":{"rendered":"Predicting the \u2018holy grail\u2019 of climate forecasting: A new model and a new public dataset"},"content":{"rendered":"

\"\"<\/p>\n

It was crunch time, just as it had been many times before in the preceding weeks. Such is the nature of real-time competition. The yearlong Subseasonal Climate Forecast Rodeo (opens in new tab)<\/span><\/a> was being sponsored by the Bureau of Reclamation and the National Oceanic and Atmospheric Administration, and teams were tasked with predicting temperature and precipitation in the Western United States for two time spans\u2014two to four weeks out and four to six weeks out. Teams were required to submit their four predictions every two weeks.<\/p>\n

Waiting until the day before the submission deadline, as they often did to base their predictions on the most up-to-date information possible, Microsoft Researcher Lester Mackey<\/a> and his teammates pulled up their data source and found\u2014nothing. The data wasn\u2019t there.<\/p>\n

It took some time to reach competition organizers, who told them the data had been moved and provided a new link, but any hope of an extension was extinguished upon making contact. Organizers reminded the group that missing and incomplete information is among the challenges facing real-world forecasters. (I mean, how often have you seen your local meteorologist skip a prediction because of insufficient information?) The disappearing act set the team back 12 hours.<\/p>\n

\u201cThat was a painful day,\u201d recalls Mackey. \u201cWe submitted something. It wasn\u2019t very good.\u201d<\/p>\n

Data is the driving force behind many research areas, the engine that powers progress, especially in statistical machine learning. This challenge, though, was unique in that data collection was mainly the responsibility of the teams, organizers providing only the sources they would use for judging the competition. Mackey and his teammates scoured different sources, including government websites, compiling the variables they thought were most important. There were quite a few times when the data wasn\u2019t released on schedule or at all.<\/p>\n

Despite the obstacles, Mackey and his teammates\u2014climatologist Judah Cohen (opens in new tab)<\/span><\/a> of Atmospheric and Environmental Research, a Verisk business; professor and consulting researcher Ernest Fraenkel (opens in new tab)<\/span><\/a>; and graduate students Jessica Hwang and Paulo Orenstein (opens in new tab)<\/span><\/a>\u2014excelled. They placed first (opens in new tab)<\/span><\/a> in predicting temperature for Weeks 3 and 4 and second for predicting precipitation for Weeks 5 and 6 in the competition, developing not only a machine learning-based forecasting system but also a comprehensive dataset for training subseasonal forecasting models. The dataset is available now via the Harvard Dataverse (opens in new tab)<\/span><\/a>. The group\u2019s work is presented in the paper \u201cImproving Subseasonal Forecasting in the Western U.S. with Machine Learning,<\/a>\u201d for which the team brought on engineer Karl Pfeiffer, also of Atmospheric and Environmental Research, to help with the historical evaluation.<\/p>\n

\u201cWhat has perhaps prevented computer scientists and statisticians from aggressively pursuing this problem is that there hasn\u2019t been a nice, neat, tidy dataset for someone to just download from the internet and use, so we hope that by releasing this dataset, other machine learning researchers or computer science researchers will just run with it,\u201d says Hwang.<\/p>\n

The \u2018holy grail\u2019 of forecasting<\/h3>\n

Today\u2019s forecasters are able to sufficiently predict the short term\u2014think your 10-day forecast\u2014using current atmospheric conditions and are even accurate enough in providing a sense of what we can expect long term, or more seasonally, using land and oceanic conditions. But subseasonal forecasting, which comprises the two-to-six week range? Cohen describes it as the \u201choly grail of forecasting.\u201d<\/p>\n

\u201cThat two-to-six-week time frame is kind of in no man\u2019s land; it\u2019s too long a period to have any signal from the conditions today, but it\u2019s too short to really get a signal from the earth\u2019s surface,\u201d explains the climatologist, who brought the idea of applying machine learning to the field and entering the contest to Mackey.<\/p>\n

Current models are physics-based simulators governed by a set of dynamical equations dependent on observations of the weather today, such conditions as temperature and humidity, to predict weather tomorrow, says Cohen. The trouble, among other challenges, is weather is fluid, changing across both time and space, and it\u2019s impossible to capture that continuity exactly in simulation. Instead, current simulators are approximations that grid up space and time and make generalizations about larger areas they represent.<\/p>\n

But even if we could perfectly simulate a weather model, instrument limitations remain problematic. The most precise instruments are imperfect and cannot simultaneously monitor all aspects of the weather at all locations and all times.<\/p>\n

\u201cThe physics-based models amplify error so rapidly that even a small discrepancy in the initial conditions leads to highly inaccurate predictions after 14 days,\u201d says Mackey, summarizing the fundamental issue presented by meteorologist and mathematician Edward Lorenz in the early 1960s (opens in new tab)<\/span><\/a>.<\/p>\n

And useful subseasonal forecasting systems matter. Farmers can better plan planting and harvesting; utility companies can call in necessary resources, like additional trucks, sooner; and water and fire management agencies can better address the strains of extreme weather events such as droughts, flooding, and fire. \u201cThey can\u2019t make that preparation a day in advance, but with a two-week heads-up, there are preparations they can make,\u201d says Cohen.<\/p>\n

According to Mackey and his team, over the years, there was a shift from statistical forecasting to today\u2019s more physics-based approach. But there is a missed opportunity in going solely with those strategies that, as Cohen explains, are more about solving the future dynamics of the atmosphere and less about studying what has already occurred.<\/p>\n

\u201cEven though, in theory, there is an infinite number of weather patterns, the atmosphere tends to repeat itself, and that can be exploited in making predictions,\u201d he says.<\/p>\n

Enter the data.<\/p>\n

A more statistical approach to predicting rain or shine<\/h3>\n

A large amount of high-quality historical weather data and existing computational power make the exploration of a more statistical approach to forecast modeling worthwhile, and\u2014as the team has shown\u2014merging both the physics-based and statistics-based approaches equals better predictions.<\/p>\n

\u201cMachine learning is essentially learning from experience,\u201d says Mackey. \u201cEven though we can\u2019t observe the future, we have many examples of past temperature patterns and the historical features that preceded them. Ample relevant data and the computational resources to extract meaningful inferences are what make machine learning so well-suited to the problem.\u201d<\/p>\n

The team\u2019s forecasting system combines two regression models trained on its SubseasonalRodeo (opens in new tab)<\/span><\/a> dataset. The dataset comprises a variety of weather measurements dating as far back as 1948, including temperature, precipitation, sea surface temperature, sea ice concentration, and relative humidity and pressure, collected from such sources as the National Center for Atmospheric Research and the National Oceanic and Atmospheric Administration\u2019s Climate Prediction Center and National Centers for Environmental Prediction.<\/p>\n

The team based the first of its two models\u2014a local linear regression with multitask model selection, or MultiLLR<\/em>\u2014on the presumption that not every factor in its dataset would be of equal importance at any given time throughout the year. For example, a relevant factor in the wintertime, such as sea ice concentration, might not be as useful in predicting temperature or precipitation in the summertime.<\/p>\n

\"\"

Mackey and his team use a local linear regression with multitask model selection to train one of their two climate forecasting models to evaluate the relevance of a variety of conditions in predicting future temperature and precipitation.<\/p><\/div>\n

Data used was limited to an eight-week span in any year around the day for which the prediction was being made, and through the selection process, which used a customized backward stepwise procedure, two to 13 of the most relevant features\u2014or predictors<\/em>\u2014were combined to make a forecast. Also included as a predictor was the prediction of a physics-based model, which the model being trained would take or leave depending on its veracity.<\/p>\n

The second model\u2014a multitask k<\/em>-nearest neighbor autoregression, or AutoKNN<\/em>\u2014incorporates the historical data of only the measurement being predicted, either temperature or precipitation.<\/p>\n

\"algorithm

A multitask k-nearest neighbor autoregression focuses on historical data of only the condition being predicted\u2014temperature and precipitation in the case of the Subseasonal Climate Forecast Rodeo.<\/p><\/div>\n

To make a prediction for a target date in the future, the model first identifies dates in history that bear the most similarity to the target date. Because the target has not yet been observed, the model creates an observable history for the target date based on the observations one year and 60 days through one year prior (for example, November 3, 2017, through January 1, 2018, when forecasting January 1, 2019). The model then judges similarity by comparing the history of the target with the history of each past date. If a candidate date\u2019s history proves to be good a predictor of the target date\u2019s history, then the candidate date\u2019s observed temperature or precipitation, whichever is being predicted, is adopted as proxy for the future date\u2019s.<\/p>\n

\u201cWe wanted to open up the playing field to any date that could have been relevant, and we found that, quite often, the dates selected weren\u2019t in the same months or seasons as the dates being predicted,\u201d says Mackey.<\/p>\n

For temperature, each of the 20 most similar dates produced a prediction for the target date. These 20 predictions were the variables in their regression model, which assigned a weight to each\u2014\u201cthink of it as how much I trust each of those predictions,\u201d says Mackey\u2014and then combined them to yield a single and hopefully more accurate prediction. Also, the model biased its prediction function to a time period closest to the target month-day combination, within 56 days to be exact.<\/p>\n

\u201cWe found that the optimal way to combine neighbors changes based on the time of year,\u201d says Mackey. \u201cThe rules that we were learning for combining the 20 dates together into a single prediction looked very different in the summer than in the winter, and so this locality was to capture that, to add a seasonality component to our prediction rule.\u201d<\/p>\n

Precipitation, which Lester said was a more difficult prediction task, didn\u2019t benefit from multiple \u201csimilar date\u201d variables, so the model used only one.<\/p>\n

\"\"

Distribution of the month of the most similar neighbor learned by AutoKNN<\/em> as a function of the month of the target
date. Left: Most similar neighbor for temperature, Weeks 3\u20134. Right: Most similar neighbor for precipitation, Weeks 3\u20134. For temperature, the most similar neighbor can come from any time of year. For precipitation, the researchers observe a strong seasonal pattern; the season of the most similar neighbor generally matches the season of the target date.<\/p><\/div>\n

The benefits of ensembling<\/h3>\n

While each model on its own performed better than the competition\u2019s baseline models\u2014a debiased version of the operational U.S. Climate Forecasting System (CFSv2) and a damped persistence model\u2014they tap into different components of the difficulties in subseasonal forecasting: The first model uses only recent history to make its predictions; the second model doesn\u2019t account for other factors influencing climate and weather. So the team\u2019s final forecasting model was an ensemble of the two. They averaged their normalized anomalies, resulting in a more accurate prediction.<\/p>\n

\u201cYou can think of each model as being a different expert, so it\u2019s like having a panel of experts and then you might combine their opinions,\u201d says Hwang.<\/p>\n

The team believes ensembling has great potential in this space to improve forecast accuracy\u2014or skill, as it\u2019s called in climatology\u2014and mitigate the inherent instability of the objective function. Right now, measuring forecast accuracy involves comparing a map of predictions and a map of \u201ctruths\u201d and noting their cosine similarity.<\/p>\n

In this work, the team ensembled just two models and for just a small portion of the planet. They hope to expand their work beyond the Western United States and continue to collaborate with the Bureau of Reclamation and other agencies to use statistical machine learning for social good.<\/p>\n

\u201cI think that subseasonal forecasting is fertile ground for machine learning development, and we\u2019ve just scratched the surface,\u201d says Mackey.<\/p>\n","protected":false},"excerpt":{"rendered":"

It was crunch time, just as it had been many times before in the preceding weeks. Such is the nature of real-time competition. The yearlong Subseasonal Climate Forecast Rodeo was being sponsored by the Bureau of Reclamation and the National Oceanic and Atmospheric Administration, and teams were tasked with predicting temperature and precipitation in the […]<\/p>\n","protected":false},"author":37074,"featured_media":563535,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"msr-url-field":"","msr-podcast-episode":"","msrModifiedDate":"","msrModifiedDateEnabled":false,"ep_exclude_from_search":false,"_classifai_error":"","msr-author-ordering":[],"msr_hide_image_in_river":0,"footnotes":""},"categories":[194455],"tags":[195035,213536],"research-area":[13561,13556],"msr-region":[],"msr-event-type":[],"msr-locale":[268875],"msr-post-option":[],"msr-impact-theme":[],"msr-promo-type":[],"msr-podcast-series":[],"class_list":["post-563532","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-machine-learning","tag-climate-data","tag-forecasting","msr-research-area-algorithms","msr-research-area-artificial-intelligence","msr-locale-en_us"],"msr_event_details":{"start":"","end":"","location":""},"podcast_url":"","podcast_episode":"","msr_research_lab":[199563],"msr_impact_theme":[],"related-publications":[],"related-downloads":[],"related-videos":[],"related-academic-programs":[],"related-groups":[330695],"related-projects":[],"related-events":[],"related-researchers":[],"msr_type":"Post","featured_image_thumbnail":"\"\"","byline":"","formattedDate":"March 7, 2019","formattedExcerpt":"It was crunch time, just as it had been many times before in the preceding weeks. Such is the nature of real-time competition. The yearlong Subseasonal Climate Forecast Rodeo was being sponsored by the Bureau of Reclamation and the National Oceanic and Atmospheric Administration, and…","locale":{"slug":"en_us","name":"English","native":"","english":"English"},"_links":{"self":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/563532","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/users\/37074"}],"replies":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/comments?post=563532"}],"version-history":[{"count":12,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/563532\/revisions"}],"predecessor-version":[{"id":593959,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/posts\/563532\/revisions\/593959"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/media\/563535"}],"wp:attachment":[{"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/media?parent=563532"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/categories?post=563532"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/tags?post=563532"},{"taxonomy":"msr-research-area","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/research-area?post=563532"},{"taxonomy":"msr-region","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-region?post=563532"},{"taxonomy":"msr-event-type","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-event-type?post=563532"},{"taxonomy":"msr-locale","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-locale?post=563532"},{"taxonomy":"msr-post-option","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-post-option?post=563532"},{"taxonomy":"msr-impact-theme","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-impact-theme?post=563532"},{"taxonomy":"msr-promo-type","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-promo-type?post=563532"},{"taxonomy":"msr-podcast-series","embeddable":true,"href":"https:\/\/newed.any0.dpdns.org\/en-us\/research\/wp-json\/wp\/v2\/msr-podcast-series?post=563532"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}