Correcting Extremes

Introduction

As Cumulus processes each reading from your weather station, it checks that value (and any derived from it) against the extremes currently stored in the relevant .ini files, and if necessary updates all records that are affected. Such record are maintained for current day in today.ini, for current month in month.ini, current year in year.ini, for each month in all years in monthlyalltime.ini and for since you first started using Cumulus in [alltime.ini]].

Actually, that all-time claim is not true if you have been using Cumulus software since it was first available as which all-time extremes are recorded has changed as more and more were added in different builds. This therefore means that if you have been using Cumulus for a while, the all-time records will actually be since different starting dates depending on the parameter:

• wettest month is since 5 April 2004, or whenever you installed version 1.2 or higher, however many of the earlier MX versions had bugs relating to wettest month
• highest minimum temperature is since 15 April 2004, or whenever you installed version 1.2.2 or higher
• apparent temperature, and heat index, are only since 6 April 2011 or whenever you installed version 1.9.1 or higher
• windrun all-time extremes are stored only from version 1.9.2 (released 5th October 2011) or whenever you installed it or a later version
• feels-like all-time extremes were introduced in version 3.6.0, but the calculation changed in 3.6.10, so any values recorded while you were using a version earlier than 3.6.10 are invalid
• humidex all-time only since you installed 3.7.0 - b3089 or a later version

If there is a rogue value read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into any of those extreme record files and it might also make related derived value extremes wrong as well.

Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that any all-time records recorded during the time you were using those versions are correct. Please see updates.txt which is part of each MX release distribution for brief details of the very many issues.

Correcting all-time extremes (held in alltime.ini) is easy as whenever Cumulus makes an update to that file, from version 1.8.9 onwards it logs the previous and new values to the Alltimelog.txt file. Consequently, if you detect a rogue value you can look up the latter file to determine what to reset the former file to.

Of course, it is possible that the old value in Alltimelog.txt is not appropriate, there might be another spot value that has occured since the rogue value update that did not update alltime.ini because of the rogue value. In this case, the easiest way to find the correct extreme is by looking at Cumulus 1 select-a-graphs, or MX charts if they cover the relevant period, and reading the relevant extreme there. These are also available on the sample trends web page. Remember the resolution of some graphs/charts will be to nearest minute, while a search through your monthly Standard_log_files has two disadvantages, first it is much harder to find the line of interest (although in both flavours the all-time extreme editors does allow you to load all the log files including dayfile.txt and see the relevant extreme loaded there, it does take a while to read all those files), and second those lines only record a small sample of spot values as by default they are only created every 10 minutes (so even if you choose to load all log files, it might not include the actual extreme you have missed recording).

When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into the editor (note that there are some early builds of each flavour that do not include an editor):

• select All-time records in the edit menu accessed from main screen in Cumulus 1,
• select All Time records in the edit menu of MX admin interface.

The way the editor allows you to change a value depends on which flavour you are using:

• Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across). Press Save button to retain the change and exit.
• You have to select either a value or a time-stamp that you want to change before you can type in a replacement value (you have to type the new number in even if it is displayed by loading the log files). Then click the tick to retain the change, and that will allow you to select another value or time-stamp to edit, before you finally exit the editor.

Please remember this edit has not affected any lines in your monthly Standard_log_files or in daily summary log, and if you are using MX, it has not affected any tables on your database server. Equally, this editor does not update any related extreme entries in other record files, you have to select this month, or this year editor subsequently to make your edit there.

How do I correct my monthly all-time records?

From version 1.9.3, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'. The release announcement did explain how to initilise these records that are held in monthlyalltime.ini. Here the old and new values can be looked up in the monthlyalltimelog.txt file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in Diags_folder.

In many respects, the instruction for the all-time editing above also apply to monthly all-time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.

Another way to hunt for former values

In the examples above, it was mentioned that all the changes to extremes were logged somewhere. That is normally the best way to find former values.

In a few cases, an alternative is to compare the alltime.ini (or monthly equivalent) with a previous version of the same file that has been stored in a backup directory. Basicallly, Cumulus takes a backup of most of the active data files when it starts up, and also at the start of the meteorological day (just after midnight for a lot of users). The backups are kept in folders within the backup sub-folder in the Cumulus installation. So yet another alternative method would be to find the latest backup from before the error occurred, and copy the alltime.txt and/or monthlyalltime.txt file from the backup to the Cumulus data folder. Do this with Cumulus stopped.

There is more information in Category:Log_Files and the pages relating to individual files, for all of the extreme holding files.

Some definitions

Rogue value

In this article, the term rogue value is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumuluative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be progated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.

Here are a typical examples:

• it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
• perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumuluative a series of wrong date points
• an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong