Correcting Extremes

This page brings together text that was originally on other pages. It is designed to cover both and , but do be aware that some terminology varies between the two flavours.

Introduction

As Cumulus processes each reading from your weather station, it checks that value (and any derived from it) against the extremes currently stored in various .ini files, and if necessary updates the extreme records that are affected.

All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered amongst pages covering the various files).

The extreme records that are maintained in this way are:

Period	File storing extremes	How to correct	Link to web tag section	Notes
For current day so far	today.ini	Editor for "Today's rain" (no editor for other derivatives)	today.htm	Many entries in this file (for non-midnight rollover, use is made of yesterday.ini too) get transferred to dayfile.txt at end of day.
For past days	dayfile.txt	See Amending dayfile	Web tags only exist for yesterday	Often used as source for corrections - see #Accuracy Note
For current month-to-date	month.ini	Editor for "This month's records"	thismonth.htm	Please see #Accuracy Note
monthyyyyMM.ini are archived copies for past months
For current year-to-date	year.ini	Editor for "This year's records"	thisyear.htm	Please see #Accuracy Note
yearyyyy.ini are archived copies for past years
For all readings since a 'start date	alltime.ini	Editor for "All Time Records"	records.htm	See table below for start date
For a particular month in all years	monthlyalltime.ini	Editor for "Monthly Records"	monthlyrecord.htm

Explaining columns in above table:

1. The first (label) column is self-explanatory
2. The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
3. The third column gives the name for the Edit menu item to choose to edit these extreme records
4. The links in fourth column leads you to more information about the web tags associated with that period, you can incorporate those in your own templates.

How editing accuracy depends on source selected

The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:

1. The (hopefully more accurate) figure taken from a search for that extreme by examining all entries in the dayfile.txt for that period
2. The (usually less accurate) figure taken from a search for that extreme by examining all entries in the Standard log files for that period

Let me explain why the first returns the more accurate result (unless the dayfile.txt line, either was created with a rogue value, or has been corrupted):

• Using Standard log files as source for recalculating past extremes:
  • Let us assume you are using the default logging interval of 10 minutes
  • Unlike some other weather station software available (which logs highest and lowest since previous log), Cumulus logs spot values
  • That means the Monthly log files do not capture any extremes recorded in the time (by default 599 seconds) between logs
  • Therefore the detailed log files are not normally the most accurate source
  • Please note, this less accurate way of deducing daily extremes/totals (to update dayfile.txt) is used by Cumulus software:
    • For Legacy Cumulus 1: Create Missing in legacy dayfile editor.
    • For MX: CreateMissing.exe utility.
• Using dayfile.txt as source for recalculating past extremes
  • MX typically processes data from your weather station every second (even if you use a weather station type that only reads its sensors every 40 or 60 seconds). Cumulus 1 processes data from your weather station at intervals that vary for the different station types, but we can assume it is at least every 60 seconds.
  • Therefore extremes recorded in today.ini' (and from there into dayfile.ini) are based on the full sampling done by Cumulus
  • This means none, or very few, extremes are missed
  • In March 2021, a new utility Create Records was planned (for use with MX only), as at July 2021 no progress has been made in coding it. It appears that this utility will read dayfile.txt and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other files like monthly-all-time and all-time. Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available.

However, if you either have had an electrical supply problem or there has been corruption of data within your weather station generating a rogue value, it is possible that the files currently being used by Cumulus get corrupted. Provided you act, as soon as possible afterwards, you can make use of the backups that Cumulus takes. All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see today.ini) - either with the end of day or start of day contents. For this month and this year extreme records, it is often better to find the previous extreme by examining a backup copy, providing a true extreme has not happened since, than by using the values displayed in the editor as those might be corrupted in the same problem.

For all-time (and for MX only, monthly-all-time) extreme records, there is a log Alltimelog.txt (and for MX only,Monthlyalltimelog.txt) that logs each time the extreme is updated, so you can get an accurate previous value from there.

Arrangement of information on this page

The page continues by first describing corrections related to the penultimate entry in that table (all-time). That approach is partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier (for example an edit to all-time indicates which month (or months) you need to edit in monthly-all-time; but an edit to monthly-all-time does not help you know whether an edit is needed to all-time).

There is more information in Category:Log_Files and the pages relating to individual files, for all of the extreme holding files.

Why might your extremes need to be corrected?

Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units. In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any Extra Sensor Files, any Standard log files, in dayfile.txt, plus the multiple.

The instructions on this page will help you correct the extreme record files as there are relatively few edits to do for a single source value changing because of units, or a calibration multiplier. There is no change to when a maximum/minimum occured, nor does it affect any derived values. For example a change in units, or calibration multiplier, for temperature or wind speed, does not affect calculations for wind chill and feels like temperature. It is a bit more complex if you change a calibration offset, as that might affect threshold for calculation of some derivatives (e.g. changing an offset for temperature or wind speed also affects wind chill). It is slightly more complicated changing units for temperature (as you may have many extra temperatures).

The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see changed rainfall units example), or to use a spreadsheet (be careful not to affects dates or times) like Libre Office Calc where you can paste special a multiplier to all cells in a particular column.

If there is a rogue value read from your weather station (this could be due to noise affecting communications, or because a sensor has been knocked), it can get into the daily summary file, any of the extreme record files, and any related derived value extremes may become wrong as well.

A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.

All-time extreme functionality

For simplicity, this article will only document the development of all-time functionality, it should be obvious that for other extremes mentioned in the introduction, full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, daily extreme functionality was added first, this month/year extreme functionality followed that, and all-time was introduced before monthly-all-time. Also, this section has intentionally been kept brief, and does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.

There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in File:Changes.zip, although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within Cumulus forum announcements, but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in month.ini and year.ini depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt, for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in dayfile.txt for the whole of that period).

Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The updates.txt that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in the relevant support forum will yield more information in return for a lot more effort.

The start date referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:

Parameter	First released	First in Version	First in Build
highest/lowest apparent temperature	26 Oct 2010	1.9.1 beta	957
highest/lowest feels like temperature	24 June 2020	3.6.10	3086
highest Canadian Humidity Index (humidex)	28 Jul 2020	3.7.0	3089
highest minimum temperature	15 April 2004	1.2.2	(lost)
highest USA heat index	29 Aug 2010	1.9.0 beta	955
wettest month	5 April 2004	1.2	(lost)
highest daily wind run	3 Jul 2011	1.9.2 beta	1001

Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked (lost), and there is some vagueness in information mentioned elsewhere in this page.

Correction of All Time Extreme records

Cumulus software makes it easy to correct the all-time extremes (held in alltime.ini).

This is because whenever Cumulus makes an update to that file, from version 1.8.9 onwards, it logs the previous and new values to the Alltimelog.txt file. Consequently, if you detect a rogue value, you can look up the latter file to determine what how to revert an entry in the former file.

Alternative ways to find a replacement for a rogue value

Of course, it is possible that the old value in Alltimelog.txt is not appropriate. It might be that after the rogue value was stored in alltime.ini, a new extreme was seen, and this new extreme was different to the previous value stored in Alltimelog.txt, but it did not cause an update in alltime.ini because of the rogue value that was stored there being more extreme.

Please see #Accuracy Note for other accuracy points.

Looking at graphical representations

One way to check whether the above speculation is correct is by looking at Cumulus 1 select-a-graphs (available from version 1.2 released on 5th April 2004), or MX charts in the admin interface (available in , if you can find a plot that covers the period between when the false extreme was recorded and now you are ready to do a correction, you can look for evidence of a new extreme that has been missed. (If you only have access to your web pages, then the sample trends web page may, or may not, cover the relevant period). Some plots record values every minute, and those high resolution plots are ideal for your search.

Using the Cumulus backup

If for any reason, the Alltimelog.txt cannot be used (maybe you deleted it in error), and you are within a week of when the rogue value updated the extreme, an alternative is to compare the alltime.ini with a previous version of the same file that has been stored in a backup directory. Basicallly, Cumulus takes a backup of most of the active data files when it starts up, and also at the start of the meteorological day (just after midnight for a lot of users). The backups are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours).

If you can find the latest backup stored for a date and time from just before the alltime.txt was updated with the rogue value, look in that backup copy to see the previous value that was in the file, and use that value when you follow editing instructions below.

Searching your standard log files

Any search through your monthly Standard_log_files has two disadvantages:

1. those lines only record a small sample of spot values as by default they are only created every 10 minutes (so even if you choose to load all log files, it might not include the actual extreme you have missed recording).
2. as the files are typically large, it takes some time to load the contents of the files.

Both the original Cumulus (it is not clear from which version, it might be from 1.7.x) and MX (from version 3.2.0 build 3056) provide all time record editors. These editors have the ability (as described below) to load up both the daily summary log file and all your Standard_log_files, so you can look at the extremes that are recorded in those files, and use that information to determine what to replace your rogue value with.

The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files.

Using the provided editors

Both Cumulus 1 and MX provide an editor (at most versions of the relevant flavour, update to a new release if your build does not include an editor).

When you know what value to edit, and what new value (or are prepared to accept whatever the editor finds in a log file) to replace it, you can go into this editor:

• select All-time records in the edit menu accessed from main screen in Cumulus 1,
• select All Time records in the edit menu of MX admin interface.

The way the editor allows you to change a value depends on which flavour you are using:

• Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across). Press Save button to retain the change and exit.
• You have to select either a value or a time-stamp that you want to change before you can type in a replacement value (you have to type the new number in even if it is displayed by loading the log files). Then click the tick to retain the change, and that will allow you to select another value or time-stamp to edit, before you finally exit the editor.

Please remember any edit you make here will not affect the related extreme in your monthly-all-time extremes recorded in monthlyalltime.ini. Nor can you update Standard_log_files, or daily summary log, by any edit made here. If you are using MX, it has not affected related tables on your database server.

Actions after you have edited all-time entries

In all cases, an edit made to an entry in alltime.ini, means that you need to make the same change in monthlyalltime.ini. Look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed. How to edit the monthly-all-time extreme records are described in next section.

If you loaded up the monthly standard log files into the all-time editor, you should be able to see:

• if there is an entry for same time-stamp as that associated with the former rogue value as seen in Alltimelog.txt
• you should also be able to work out, from Alltimelog.txt, whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.

If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).

If the correction you have made is in the current month, you will also need to change the entries in month.ini and year.ini. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same data folder, with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter.

The all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.

Correction of Monthly All Time Extreme records

From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.

Initialisation of monthly-all-time extreme records

Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one Copy button, the one in the header row, all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking OK button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see Calculate Missing Values).

How do I correct my monthly all-time records?

In many respects, the instruction for the all-time editing above also apply to monthly all-time.

• If you are a Cumulus 1 user, you do not have a monthlyalltimelog.txt file to look in to see the log entry with the previous and new values. This information was logged into a file in the diagnostic files folder. If you have restarted Cumulus several times since that entry, the file may have been deleted, but take a look if a file with the right date does still exist, and then you know what value to revert to.
• If you are a MX user, see if you have a monthlyalltimelog.txt file in the same data folder, and then you will know what value to revert to.

Here the old and new values can be looked up in the monthlyalltimelog.txt file if that exists (it was introduced by MX, so it does not exist in Cumulus 1, where such changes were logged to files created in Diags_folder.

There are the same alternative ways to look for the values to enter in the monthly-all-time entries that you need to change, as were described earlier to look up value for all-time. If you have already done an all-time edit, then look at the month part of the time-stamp for the former and newer entries in all-time to tell you whether there is one (both same month) or two (former and newer are different months) changes needed for monthly-all-time.

The actual editor, for Cumulus 1 or MX, may be found from the same menu as is described above for all time. One difference obviously is that you do have to choose the tab that corresponds to the month you wish to edit. I leave you to work out any other differences.

Actions after you have edited monthly-all-time entries

If you loaded up the monthly standard log files into the monthly-all-time editor, you should be able to see:

• if there is an entry for same time-stamp as that associated with the former rogue value as seen in monthlyalltimelog.txt
• you should also be able to work out, from monthlyalltimelog.txt, whether the time recorded (for that rogue value) does correspond to a time (by default every ten minutes) when readings were stored in that file.

If you have not yet looked at the monthly standard log file, the second point may still help you to decide if you might need to edit the file (and how to do so will be explained later).

If the correction you have made is in the current month, you will also need to change the entries in month.ini and year.ini. They are relatively small files, so it should be easy to use the editors (as described later) to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same data folder, with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.

The monthly-all-time editor does not edit daily extremes, yet it is likely the change you have made will affect two lines (identified from date part of time-stamps for former and newer entries in all-time), and you will use the dayfile.txt editors described later.

Correction of extremes for today

The Cumulus Edit' menu includes a Today's rain option where you can adjust the total rainfall for today (e.g. if you or the wind have knocked your rain gauge) as described below. There is no facility provided to edit any other content of today.ini, but since today.ini is used to create lines in dayfile.txt, you can follow instructions in Amending_dayfile to make any necessary corrections for past days.

There is an error in today's total rainfall

Easy - correct today's total using the 'today's rain' editor on the edit menu.

• select 'Today's rain in the edit menu accessed from main screen in Cumulus 1,
• select Today's rain in the edit menu of MX admin interface.

This edit will actually alter the start of day rainfall counter figure:

• If you want today's rain to seem less (perhaps you or the wind knocked the rain gauge), Cumulus will increase the start of day counter
• If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter

Please note that this edit does not affect "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the Standard_log_files, your database will retain incorrect values, as these are not updated by this correction.

Please see Today.ini#Editing_rainfall_in_today.ini_within_Cumulus for details of how to edit related fields.

Correction of extremes for month-to-date

As mentioned in passing above, a rogue value may get recorded in a month.ini file.

You should ensure you have got the daily summary log (dayfile.txt) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this month.

Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit This year's records. The editors work in exactly the same way as was described for all-time above.

• In Cumulus 1, Copy' buttons enable you to copy a record from dayfile. Click OK to save.
• In MX, the dayfile value/timestamp, and the Logfile value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click 'Tick to save.

Correction of extremes for past month

• Cumulus 1 never allows you to see a month.ini file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
• From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the month.ini at the start of a new year, it saves the old month.ini (whenever it was last updated) as a file with a name like month201501.ini.

MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.

Correction of extremes for year-to-date

As mentioned in passing above, a rogue value may get recorded in a year.ini file.

You should ensure you have got the daily summary log (dayfile.txt) correct, as described in that link, before you attempt this correction, as the provided editor makes it easy to copy from dayfile to extremes for this year.

Both Cumulus 1 and MX (at most versions of the relevant flavour, update to a new release if necessary) provide an editor. The same menu mentioned above for editing all-time extreme records has an option to edit This year's records. The editors work in exactly the same way as was described for all-time above.

• In Cumulus 1, Copy' buttons enable you to copy a record from dayfile. Click OK to save.
• In MX, the dayfile value/timestamp, and the Logfile value/timestamp are shown in the editor. You click on the value or timestamp, manually overtype with new content, and click 'Tick to save.

Correction of extremes for past year

• Cumulus 1 never allows you to see a year.ini file when the year is completed, because at the end of the year it is initialised ready for the new year.
• From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the year.ini at the start of a new year, it saves the old year.ini (whenever it was last updated) as a file with a name like year2015.ini.

MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.

Some definitions

Rogue value

In this article, the term rogue value is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.

Here are a typical examples:

• it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
• perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
• an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong

Flavour, Release, Version, and Build

Flavour is used to represent the original Cumulus, Cumulus 2, and Cumulus MX, collectively. Where the text says applicability is dependant on flavour, it means that the action you do depends on whether you are installing/running Cumulus MX or the original Cumulus software.

Release is used to signify what the Cumulus developer makes available for download after there has been a modification to the software. For most Cumulus 1 builds, the release consisted of a executable that would create all the folders and files needed to run that software. For Cumulus 2, releases were a zip file and were numbered using a identifier with many digits. MX releases are as a zip file that is labelled with the build number.

Version here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.

Build number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number. For recent MX releases, the developer has changed version number every time there is a new build released.