Correcting Extremes

This page brings together text that was originally on other pages. It is designed to cover both and , but do be aware that some terminology varies between the two flavours.

Introduction

As Cumulus processes each reading from your weather station, it checks that value (and any derived from it) against the extremes currently stored in various .ini files, and if necessary updates the extreme records that are affected.

All too often a mistake in one extreme record is propagated to other extreme records, so the purpose of this page is to cover all the necessary corrections in one place (previously the information was scattered amongst pages covering the various files).

The extreme records that are maintained in this way are:

Period	File storing extremes	How to correct	Link to web tag section	Notes
For current day so far	today.ini	Editor for "Today's rain" (no editor for other derivatives)	today.htm	Many entries in this file (for non-midnight rollover, use is made of yesterday.ini too) get transferred to dayfile.txt at end of day.
For past days	dayfile.txt	See Amending dayfile	Web tags only exist for yesterday	Often used as source for corrections - see Depends on source selected Note
For current month-to-date	month.ini	Editor for "This month's records"	thismonth.htm	Please see #Accuracy Note
monthyyyyMM.ini are archived copies for past months
For current year-to-date	year.ini	Editor for "This year's records"	thisyear.htm	Please see Accuracy Note
yearyyyy.ini are archived copies for past years
For all readings since a 'start date	alltime.ini	Editor for "All Time Records"	records.htm	See table below for start date
For a particular month in all years	monthlyalltime.ini	Editor for "Monthly Records"	monthlyrecord.htm	Similar to previous row, but different start date, and separate extremes maintained for each month (regardless of the year)

Explaining columns in above table:

1. The first (label) column is self-explanatory
2. The second column contains a link to the page that explains more about the file named there, which is where the extreme records are stored for that period
3. The third column gives the name for the Edit menu item to choose to edit these extreme records
4. The links in fourth column leads you to more information about the web tags associated with that period, you can incorporate those in your own templates.

General Editing Advice

This Wiki page talks about rogue values, meaning any incorrect entry in one or more of the extreme record files. You may have a feel as to which files in above table will need correction, but generally it is best to start with all-time and finish with daily extreme records, not the reverse order.

The remainder of this Wiki page describes general techniques for correcting rogue values, including using the built-in-editors, and gives guidance on all the different ways to find correct values.

It is highly recommended that you always start your extreme record correction by seeing if the error is present in the Alltime.ini file that holds all-time extreme records. That approach is best partly because many Cumulus Users take a lot of interest when their all-time extreme records are broken, and partly as all-time is a good place to start as it can make subsequent edits easier. For the all-time extreme records, each individual update is logged in Alltimelog.txt from version 1.8.9 onwards. You may get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded. If the error has affected all-time, looking at the tracking log tells you whether the error will have also affected the relevant month monthly-all-time, and whether it has affected this month/year, so you have pointers to what other files to edit.

If the rogue value has not affected the all-time extreme records, it is recommended you see if the error is present in the monthly-all-time file.

• From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.
• If you are using Cumulus 1, then make the best guess as to which tab to pick, or work through each tab until you find the month affected.
• If you use MX, then Monthlyalltimelog.txt) logs each time any extreme is updated, so that file tells you which tab has the rogue value. You may also get an accurate previous value from this tracking log, it depends on sequence of extreme values, the value you want may not have been noted if the rogue extreme occurred before the value you want, so stopped any subsequent true extreme being recorded.

Again, finding the rogue value has affected monthly-all-time gives you a date, that is a steer to whether the this month extremes file will need correction, and whether this year extreme records file will need correction. They are relatively small files, so it should be easy to use the built-in editors to edit them. If you are a Cumulus MX user, then old versions of those two files, for past months and past years, are retained in the same data folder, with the relevant date added as a suffix, so although MX does not provide an editor for them, you may want to use a standard text editor to amend the relevant parameter in a file called something like month202001.ini', where the final 2 digits correspond to the month tab (or month tabs) you have just edited. Similarly check any old year file and see if you need to edit it.

In working through the various files in above table, remember that if the rogue value was recorded today, then today.ini will be wrong:

• In Cumulus 1, you can edit today.ini without stopping the software, provided you get the timing right, but it is more sensible to stop Cumulus before editing that file
• In MX, the values are held internally, with periodic updates to today.ini, so any edit you make to that file while MX is running is ignored. Since MX does not provide a today.ini editor, you must stop MX (see MX on Linux or MX on Windows OS) and edit the file using a text editor or programmer's editor that will not add unwanted content to the file.

If the rogue value relates to yesterday, then you must edit dayfile.txt, and you may choose to update yesterday.ini although this latter file will be overwritten at next rollover.

Correcting multiple extremes

There are two cases to consider:

Correcting all extremes

Sometimes a mistake is made in setting up or calibrating a sensor, or (despite the warnings within both flavours of Cumulus about getting your choice of units correct from the start), you decide to change your units.

In both cases, you will identify a constant/multiplier adjustment to be applied to adjust all values (luckily times and dates of extremes are not affected) over the past period. In both cases, you need to correct past entries in any Extra Sensor Files, any Standard log files, in dayfile.txt, plus the multiple.

The built-in editors only correct one extreme record at a time, so they cannot be used for such a task.

It is important to remember that there are source and derived values in Cumulus. If you change the units, or introduce a calibration multiplier/offset, that affects the source values, but as derived values are calculated from spot values (e.g. temperature, wind speed, humidity, all recorded at same time), you cannot simply change extremes for derived values by any constant/multiplier. Please see Calculate Missing Values page for further advice.

The easiest way to change entries in any Extra Sensor Files, any Standard log files, and in dayfile.txt, is to either write a batch editing script (see changed rainfall units example), or to use a spreadsheet (be careful not to affects dates or times) like Libre Office Calc where you can paste special a multiplier to all cells in a particular column.

Correcting extremes within a period

There are a further two sub-cases that fall in this category. Both are near impossible to resolve!

Both Cumulus 1 and MX have had bugs in some releases of their software. This may mean that some of the past extremes need correction because incorrect calculations were used when those extremes were recorded, it is not possible here to say exactly how to correct these, but essentially extremes can only be recalculated from corrected spot values, and all the files for that past time will have incorrect data, so any correction is likely to be a slow extremely complex process!

There were bugs introduced sometimes in builds of the original Cumulus (known now as legacy Cumulus 1). Information about a few of the bugs and fixes can be found in File:Changes.zip, although that does not cover any 1.7.x versions, nor does not detail bugs created (and fixed) within the beta builds. More information may be found by searching within Cumulus forum announcements, but it will require a lot of effort (as there are a lot of posts to search). (For historic interest only, one example is that what is stored in month.ini and year.ini depends on when they were first created, because they are initiated from the daily summary log, dayfile.txt, for the relevant period. Therefore, an individual parameter can only be initialised if the corresponding field is present in dayfile.txt for the whole of that period).

Cumulus MX had lots of bugs in its early builds. So if you ever used Cumulus MX versions 3.0.0 to 3.3.0, you cannot rely that all all-time extreme records reported correctly take into account any records broken on a date prior to 19 Feb 2020. Also there have been some changes in how some derivatives are calculated, and these might invalidate other 2020 dated entries. The updates.txt that is part of each MX release distribution has brief details of when the very many issues were fixed. Again, searching all the posts in the relevant support forum will yield more information in return for a lot more effort.

A sensor might fail, and Cumulus does not recognise that "Null" (this might mean the weather station sends all bits set to zero or all bits set to one) values should be ignored when comparing against existing extreme records, and so set the extreme record to zero or maximum number that the number of bits can convey.

In this second sub-case, you again effectively have rogue measurements over an extended past period. Theoretically you can correct using a special batch editing script, or an external editor, but in this case you have to decide what value to use to represent not-working. You don't want to use a value that affects extremes (so you can't use an obviously wrong high or low value), you can't blank off any extreme (set it to empty), and Cumulus will not accept "--" type inputs, or anything else that might represent a null. Some people take values from a neighbouring measuring station or in some other way insert values that are good approximations. However, there is no official solution to this problem!

Built in extreme record editors

Cumulus 1 gained extreme record editing functionality from version 1.9.1 6th April 2011. See screenshot Edit menu for how to select which file to edit, once on required editing screen, follow instructions on screen:

• Simply type over the existing value and time-stamp as shown (or if you have loaded the log files, select which value to copy across and click "Copy" to copy figure from identified log file to extreme record file). Press Save button to retain the change and exit.

Cumulus MX gradually gained extreme record editing functionality in releases from 3.1.1 - b3054 to 3.4.0 - b3064 (1 Nov 2019 to 19 Feb 2020), with a major redesign of user interface in release 3.18.0 (b.3189 14 June 2022):

• In latest MX release, if you click on an individual extreme record, then a pop-up appears where you can directly edit value and time-stamp to typed new content
• To load data from Standard log files you have to click on a button, from release 3.2.0 - build 3056 - only the relevant dayfile.txt entries are shown by default
• In latest MX release, if you click on a value in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy value across or not, this does not update date/time
• In latest MX release, if you click on a date/time-stamp in either dayfile.txt or standard monthly log file columns, then you are presented with yes/no options to select to copy date/time across or not, this does not update value
• There is some validation on editing, you cannot empty the content of any extreme record

The built-in editors do allow you to load the standard monthly log files as well as the daily summary file (dayfile.txt). The other advantage of loading up these log files is that it allows you to see if the rogue value is also present in these log files. However, do exercise caution about using values from such sources, because as will be explained next, they may not hold the extreme values.

How editing accuracy depends on source selected

The editors built into Cumulus, for long term extremes (over a period of a month or more), give you the ability to display, for each extreme record:

1. The (hopefully more accurate) figure taken from a search for that extreme by examining all entries in the dayfile.txt for that period
2. The (usually less accurate) figure taken from a search for that extreme by examining all entries in the Standard log files for that period

Obviously, it is possible that the dayfile.txt file has been corrupted, and if you are using MX, then you may want to read about a utility that can recreate dayfile.txt.

But generally, dayfile.txt is the best starting point for recreating any longer period extreme records, to understand why read on:

• Using Standard log files as source for recalculating past extremes:
  • Let us assume you are using the default logging interval of 10 minutes
  • Unlike some other weather station software available (which logs highest and lowest since previous log), Cumulus logs spot values
  • That means the Monthly log files do not capture any extremes recorded in the time (by default 599 seconds) between logs
  • Therefore the detailed log files are not normally the most accurate source
  • Please note, this less accurate way of deducing daily extremes/totals (to update dayfile.txt) is used by Cumulus software:
    • For Legacy Cumulus 1: Create Missing in legacy dayfile editor.
    • For MX: CreateMissing.exe utility.
• Using dayfile.txt as source for recalculating past extremes
  • MX typically processes data from your weather station every second (even if you use a weather station type that only reads its sensors every 40 or 60 seconds). Cumulus 1 processes data from your weather station at intervals that vary for the different station types, but we can assume it is at least every 60 seconds.
  • Therefore extremes recorded in today.ini' (and from there into dayfile.ini) are based on the full sampling done by Cumulus
  • This means none, or very few, extremes are missed
  • In March 2021, a new utility Create Records was planned (for use with MX only), as at July 2021 no progress has been made in coding it. It appears that this utility will read dayfile.txt and use the more accurate daily extremes it finds there, as a basis for updating longer term extremes in the other files like monthly-all-time and all-time. Perhaps you my reader can be the contributor who updates this if the proposed utility becomes available.

Cautionary warning and other files to examine

File corruption can happen as a response to an electrical supply problem, a temporary input/output problem within the operating system, or the storage device being used, as well as after corruption of data within your weather station generating a rogue value fed through to current files.

Any or all of the Extreme Record .ini Files currently in use by Cumulus may get corrupted, as Cumulus gains exclusive write access that can overwrite the entire file during an update.

Remember the values displayed in the built-in editor from dayfile.txt or monthly standard log files just might have been corrupted in the same problem. Cumulus only appends new lines to the end of these files, so it should never overwrite the whole file, but it is possible for a connection problem to make Cumulus start a new file.

Therefore it is worth noting where else you can look to find values and date/time-stamps to use when correcting rogue data, and the next few subsections make some suggestions.

Update tracking logs

Cumulus 1 logs most extreme updates in files stored in Diags folder. Read that cross-reference for more guidance. As already mentioned, there is a log Alltimelog.txt that tracks the updates to all-time extremes.

Cumulus MX can log useful information in MXdiags folder, depending on settings mentioned in that cross-reference, but it maintains two logs Alltimelog.txt and Monthlyalltimelog.txt as already mentioned.

These tracking logs can, in certain circumstances, be the best place to look up previous values as replacements for rogue values, when the built-in editors reveal that the rogue values exist in dayfile.txt and (possibly) the standard monthly log, so you can't within the editors find the correct value you seek.

Looking at graphical representations

Many people find it easy to interpolate replacements for rogue values by looking at graphical representations of their weather data covering before and after the time when the rogue figure got recorded.

In Cumulus 1, the obvious place to look is select-a-graphs (available from version 1.2 released on 5th April 2004).

In Cumulus MX, later releases also have a select a chart feature, that may be more useful than the standard charts (in both interface web pages and default web pages).

Some plots record values every minute, and those high resolution plots are ideal for your search.

Using the Cumulus backup

Cumulus makes backups of the extreme record files that are kept in folders within the backup sub-folder in the Cumulus installation, with a maximum of about 8 being retained (it varies between flavours), so this method cannot be used for rogue values that are a week old or older.

If you notice the rogue update when it happens, remember provided you act, as soon as possible afterwards, you may be able to make use of the earlier version of the relevant extreme records file, as a source of correct extreme records before the corruption by a rogue figure.

All the extreme record files mentioned in the table above are backed up when Cumulus is restarted and (depending on which release you are using - see today.ini), with their contents just as they were either with the end of day or start of day. It is therefore possible no true extreme has happened since the most recent backup, or maybe by comparing two recent back-ups you can obtain guidance on when the last true extreme occurred. Obviously, such back-up files are no use for correcting daily extremes, but for this month, monthly-all-time, this year, and all-time, extreme records, updates to extremes don't always happen every day, especially when near end of a month. Therefore there is a good chance that you can find the previous extreme by examining a backup copy, providing a true extreme has not happened since.

Searching recent history

Cumulus 1 only provides one way to access the Recent_History, and that is by web tags. It is not easy, but if you know the time when a rogue value was reported, it may be possible to check values slightly earlier and slightly later by requesting them using web tags.

Cumulus MX stores its Recent history in a SQLite3 database that you can read. To avoid making this page too technical, it won't tell you what software can read that database, but if you have the technical knowledge, you can look up details of suitable packages to install to read the entries in the database, and perhaps find correct extremes recorded there, as that for the last week stores measurements at (normally) one minute resolution.

Extreme monitoring start-dates

As Cumulus has developed, various derived values have been added that it calculates in addition to whatever your weather station supplies. At some releases, these extras are only available via web tags for current values, and it may be some significant time later that a release makes them available as all-time, or other period, extremes. You may be able to track these changes by examining "changes.txt" for Cumulus 1 or "updates.txt" for MX, but those sources are not comprehensive.

There is no guarantee that this Wiki content has been checked, or that it is up to date. Any contributor is welcome to make corrections or bring it up to date

For simplicity, this article will only document the development of yearly functionality and attempts to record when various extreme records became available in that context.

It should be obvious that full extreme record data was not available in all Cumulus releases for all the weather variables that the latest release reports. In general, for Cumulus 1, daily extreme functionality was added first, then all-time, followed by this month/year extreme functionality, and finally monthly-all-time. For MX, generally extras were added as current values first, and later the extremes for all the various periods were added together, but development does not always happen in a consistent way!

This section has intentionally been kept brief, so it does not list all bugs that might result in incorrect extremes being stored, nor when such bugs were subsequently resolved.

The start date referenced in the last bullet in the introduction, is generally when you first started using Cumulus. However, as Cumulus has developed it has started monitoring more extreme records compared to those it was previously monitoring, so if you were using Cumulus software before 28 Jul 2020, you should check the following table. For any parameter you select in the table, the monitoring of all-time extreme records started whenever you decided to install the release shown in the following table, or a later release:

Parameter	First released	First in Version	First in Build
highest/lowest apparent temperature	26 Oct 2010	1.9.1 beta	957
highest/lowest feels like temperature	24 June 2020	3.6.10	3086
highest Canadian Humidity Index (humidex)	28 Jul 2020	3.7.0	3089
highest minimum temperature	15 April 2004	1.2.2	(lost)
highest USA heat index	29 Aug 2010	1.9.0 beta	955
wettest month	5 April 2004	1.2	(lost)
24 hour rainfall	30 Apr 2022	3.16.0	3182
highest daily wind run	3 Jul 2011	1.9.2 beta	1001
Sunshine hours	31 July 2021	3.12.0	3134

Please note the Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software on 2 Jun 2008 without preserving what had existed before. This was some months before key information in the forum started being copied to this Cumulus Wiki. Consequently, all his announcements prior to that were lost, this is why some details in above table are marked (lost), and there is some vagueness in information mentioned elsewhere in this page.

Correction of Monthly All Time Extreme records

From version 1.9.3 beta build 1033 released on 10 April 2012, Cumulus introduced the ability to monitor extremes like 'highest ever January temperature'.

Initialisation of monthly-all-time extreme records

Although the release did not automatically initialise monthly-all-time extreme records, the new monthly records editor provided in that release had a "fetch dayfile" button. By clicking just one Copy button, the one in the header row, all the relevant daily records were copied into the monthly-all-time records for the month of the selected tab. Therefore by doing that again for every other tab (except any tab for a month when you had never used the original Cumulus), and then clicking OK button, you manually initialised all the parameters (assuming your dayfile had all the parameters - see Calculate Missing Values).

Correction of extremes for today

The Cumulus Edit' menu includes a Today's rain option where you can adjust the total rainfall for today (e.g. if you or the wind have knocked your rain gauge) as described below. There is no facility provided to edit any other content of today.ini, but since today.ini is used to create lines in dayfile.txt, you can follow instructions in Amending_dayfile to make any necessary corrections for past days.

There is an error in today's total rainfall

Easy - correct today's total using the 'today's rain' editor on the edit menu.

• select 'Today's rain in the edit menu accessed from main screen in Cumulus 1,
• select Today's rain in the edit menu of MX admin interface.

This edit will actually alter the start of day rainfall counter figure:

• If you want today's rain to seem less (perhaps you or the wind knocked the rain gauge), Cumulus will increase the start of day counter
• If you want today's rain to seem greater (perhaps the rain guage got blocked by a leaf), Cumulus will decrease the start of day counter

Please note that this edit does not affect "rain rate", "maximum hourly rain", "maximum 24-hour rain", or "rain since midnight", nor does it update every data log line so it has correct rainfall counter reading. Also, if you ask MX to automatically insert a new row into a monthly table on your database server whenever a new line is stored in the Standard_log_files, your database will retain incorrect values, as these are not updated by this correction.

Please see Today.ini#Editing_rainfall_in_today.ini_within_Cumulus for details of how to edit related fields.

Correction of extremes for past month

• Cumulus 1 never allows you to see a month.ini file when the month is completed, because at the end of the month it is re-initialised ready for the new month.
• From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the month.ini at the start of a new year, it saves the old month.ini (whenever it was last updated) as a file with a name like month201501.ini.

MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.

Correction of extremes for past year

• Cumulus 1 never allows you to see a year.ini file when the year is completed, because at the end of the year it is initialised ready for the new year.
• From build 3035 released 2 Dec 2015, before the MX beta (3.0.0) overwrites the year.ini at the start of a new year, it saves the old year.ini (whenever it was last updated) as a file with a name like year2015.ini.

MX does not provide any functionality to read this old file, let alone edit it. However, if you were to edit it outside Cumulus, you probably have done an edit to either alltime.ini or monthlyalltime.ini and know what old value is wrong, and what should the value for future.

Some definitions

Rogue value

In this article, the term rogue value is used for when in Cumulus you see a value that you believe should not be there. Generally, it refers to a single data point, but where that weather derivative is cumulative in nature it might affect a string of recorded values. Regardless of whether it is single or not, such a rogue value can be propagated into several of the extreme derivatives that Cumulus calculates and maintains in its various logging files.

Here are a typical examples:

• it might appear that a gust of 89 mph was recorded as the highest on a day when you are sure it was not that windy, a single data point is wrong
• perhaps you saw 478.8mm of rain occurring on a dry day, this might be a single data point error, or as rain total is cumulative a series of wrong date points
• an extreme can be attributed to wrong time (or even wrong day), because the time on your weather station clock is wrong

Flavour, Release, Version, and Build

Flavour is used to represent the original Cumulus, Cumulus 2, and Cumulus MX, collectively. Where the text says applicability is dependant on flavour, it means that the action you do depends on whether you are installing/running Cumulus MX or the original Cumulus software.

Release is used to signify what the Cumulus developer makes available for download after there has been a modification to the software. For most Cumulus 1 builds, the release consisted of a executable that would create all the folders and files needed to run that software. For Cumulus 2, releases were a zip file and were numbered using a identifier with many digits. MX releases are as a zip file that is labelled with the build number.

Version here is a precise term, it identifies collectively all builds that are given a particular version number, that can include alpha and beta releases. For Cumulus 2, the log string of digits that identifies each release was sometimes called the version number. For the original Cumulus, and some MX releases, the version number only changes when new features are included in the release. Major functionally changes affect digit after the first decimal point (digit before decimal point identifies the flavour), while for minor functionally changes, a third part is added to the version number.

Build number in Cumulus 1 and 3 (MX), was used to identify each release, and historically alpha, beta, and bug fixing, releases could all share the same version number.

Steve Loft generally went through a lot of beta releases identified by build number before finally having a stable release with new version number. Most beta releases were available to everyone.

Mark Crossley has sometimes issued his beta commits identified by beta, beta 1, beta 2, etc. without changing build number, and sometimes incremented build number several times between changes to release version number. Most beta releases are made available to just a select few testers, via a sub-forum with limited access, or via direct email of zip files to particular people. The beta history is not documented in "updates.txt", which only quotes the more significant changes.