Recent history

This page is the new Recent History page. This page is part of a complete rewrite of the Webtags page. As long as this message exists please refer to Recent History (preserving history).

This page is Work in Progress until this message is deleted.

If any questions or remarks please send a PM to HansR on the forum. Do not use the discussion tab unless accompanied by a PM: there is no automated signal to HansR if you use that page.

This page and its subpages will document the Webtags from the point of view of the CumulusMX software and not the Cumulus 1 legacy software. Cumulus 1 is no longer maintained and will eventually disappear. If you are still using it, please refer to the original Webtags page where @sfws has tried to give both packages equal value. That effort is no longer maintained in this page. We hope you will understand this approach.

Please do not edit this page while it is being worked on i.e. while this message is still here.

---

Recent History - Introduction

Recent history functionality in Cumulus is the ability to store weather values, for the last 7 days.

Only some spot readings are stored in Recent History, the stored values will miss any extremes that occur between the times when the recent history values are stored. The weather value that tends to vary the most is wind speed, so that is where peaks are most often missed.

When Cumulus is running normally, entries are stored once a minute, regardless of how often Cumulus interrogates the weather station (might be every 10 seconds, or more or less frequently) and regardless of how often the weather station gets updated readings from sensors (might be every few seconds, every 40 seconds, every minute or less frequently). This can mean that the readings stored on a 1-minute basis may actually be duplicates if the station has not supplied a new value since the previous minute, or may not capture every reading from sensors if they are read more than once in a minute.

When Cumulus is restarted, if the weather station provides historical catch-up data, then the interval between entries for that period when Cumulus was not running, is stored at whatever interval the historical catch-up data uses. Entries less than a week old, from when MX was previously running will be retained, and those will be at one minute interval.

The data are held in RecentData table in cumulusmx.db (uses SQLite). There are many software tools available that can read/edit SQLite3, this is not the place to get too technical, but take a look at sqlitebrowser in a Windows and Linux environment, and phpLiteAdmin in Microsoft Windows. Also, you could use any application that can read ODBC files (e.g. Libre Office's Base functionality) can read the MX databases.

While Cumulus is left running, every minute:

1. The oldest set of records, in database table row number 0
2. Cumulus then works through subsequent table rows (or array elements), as it reads each one, it moves it to the previous (now empty) row (or array element)
3. The number of table rows, or array elements, is actually variable, this is because of two reasons:
   • Table rows, or array elements, will be less frequent (or even missing) for the period before Cumulus is started
   • Cumulus uses current time, read from the computer, in its decision as to where to store each new recent history value set, so it can overwrite existing table rows, or array elements, if date-time matches one that already exists, such as in the hour after when clocks go back.

Using Recent History

By a request through a Webtag to Cumulus specifying 1 to 10 079 minutes (equals 7 days) ago. Then Cumulus will retrieve the value for the nearest available time.

The normal way to retrieve a specific value is by using Recent History Webtag names and specifying how much time you want before the now using the parameters.

You'll get the nearest value if you ask for a time for which there is currently no exact match, and the Timestamp tag tells you that nearest time.

There are a number of internal calculations, such as calculating Average temperature, Heat/cold degree days and Chill hours, which use the recent history data stored in the Recent History Database for CumulusMX. These data are also used for the standard charts and the JSON data for the charts on the standard website and many other websites. See: Uploading data to a webserver outside CumulusMX .

Which weather values are stored

For Current Conditions, Cumulus can display values from the basic set of weather sensors it expects, from some derived values, and from various extra sensors, as per table below.

As Cumulus has been developed, more and more of these have been stored for Recent History. Those available in current version of CumulusMX release are listed at RecentData table in cumulusmx.db.

Warning when Daylight Saving Time starts or ends

Please note that parameters specify time-stamped array element to retrieve based on counting back from current local time so the result for any period including when clocks change may not be quite what you anticipated.

Changing from Summer to Winter time

Remember, when clocks go back, that for a whole hour following the clock change, clock times (now in winter time) repeat the previous clock times (that were in summer time). The way that Cumulus has coded this functionality, instead of the new sets of spot values being added after the old rows/elements, they overwrite the old table rows, or array elements, for the same times. Consequently, you cannot retrieve any values for the times that have been overwritten, and you must be careful when you specify to Cumulus which values you wish to retrieve.

Changing from Winter to Summer time

As for when the clocks go forward, the new values do get added after the old rows/elements, but of course there will be one hour's worth of times that simply do not exist. Consequently, any attempt to retrieve values around the time of the clock change has to be careful not to specify any time in that non-existent period, as Cumulus will just return the value before, or the value after, depending on which is nearest.

Technical note

As explained earlier, you request an entry from Recent History by specifying 1 to 10 079 minutes ago. Cumulus looks at current clock time, calculates what time would be that long ago, and seeks the nearest datetime entry.

Since SQLite database tables use the row number as the primary key (see cumulusmx.db), the datetime column does not have to contain a unique value, so technically it would be possible for MX to create rows in the database that have same time as other rows when the clocks go back; equally technically when clocks go forward, it would technically be possible to calculate a revised timestamp so you don't have 60 minute specifiers for which the time never existed.

However, to do this would involve more complicated code, because MX would need to keep track of which row was last updated, keep track of when the clocks changed, and modify its search routine to find the correct row. Life is much easier if the Cumulus user makes allowance for any clock change when making their request.

If the derivative you want is not available in your Cumulus release

It was mentioned earlier that more recent history web tag names have been added as Cumulus developed. In many cases, Cumulus has actually started calculating the derivative and using it for charts, in earlier releases than when it becomes available for others to use via web tags.

From 3.12.0, the availability of the database for viewing outside Cumulus, means that anyone who knows how to read a SQLite database can write a script, or program, to access any derivative that is in the database, but not available as a web tag, and can then pass the output to their web site by an appropriate technical process.

Alternatively, although Humidex, 'Apparent Temperature', 'Feels Like temperature', and other derivatives Cumulus can calculate, are not available in all releases, they can be calculated in a script from recent 'outside temperature', 'wind speed', and 'relative humidity' values (using the same time selection for all).

There are other derivatives that can be calculated similarly from a set of simultaneous values. Note that Cumulus 1 and MX do not always use identical formula, and although MX added Feels Like it has changed the formula a few times.

The relevant formulae using JavaScript, you need to adjust them for other languages, for some of these are shown below:

Canadian Humidity Index

If you are in USA and use Fahrenheit instead of Celsius, you will need to omit the 5/9 term, but as the index is dimensionless no other conversion is needed. This example is for 3 hours ago, change the input modification parameter to suit your need.

Cumulus 1:

H = <#RecentOutsideTemp h=3> + 5/9 * (6.1094 * Math.exp(5417.753 *(1/273.16 - 1/ (273.16 + <#RecentDewPoint h=3> )))-10);

Cumulus MX:

svp = 6.112 * Math.exp((17.62 * <#RecentOutsideTemp h=3) / (243.12 + parseFloat(<#RecentOutsideTemp h=3))); H = (5/9 * (<#RecentHumidity h=3> /100 * svp - 10)) + <#RecentOutsideTemp h=3;

Apparent Temperature and Feels Like

Note this apparent temperature formula uses Celsius for temperature and metres per second for wind speed. You will need to do the appropriate conversions from the quoted recent history tags if you use different units. The Australian Apparent temperature formula is same for Cumulus 1 and MX:

var actualVaporPress = <#RecentHumidity h=3>/100) * 6.105 * Math.exp(17.27 * <#RecentOutsideTemp h=3>) / (237.7 + parseFloat(<#RecentOutsideTemp h=3>)))); var appTempDegC = parseFloat(<#RecentOutsideTemp h=3) + (0.33 * actualVaporPress) - (0.7 * <#RecentWindSpeed h=3>) - 4;

Feels Like was implemented as a recent history web tag at version 3.6.11 (see Feels Like section below Current condition web tags) for the gradual introduction of feels like elsewhere. For earlier MX versions, and if you are using Cumulus 1, you can calculate it:

The formulas below use Celsius for temperature and km per hour for wind speed. Again, you will need to do the appropriate conversions from the quoted recent history tags if you use different units.

Calculation from recent history tags is much more complicated because there are 3 different calculations: Feels Like reports exactly same as wind chill for temperatures below 10°C or 50°F so the WC here should equal <#RecentWindChill h=3>:

if(<#RecentWindSpeed h=3> < 4.828) 	WC =  <#RecentOutsideTemp h=3>; 
else{
	wind_pow =  Math.pow(<#RecentWindSpeed h=3>,	 0.16);
	WC = (13.12 + 0.1625 * <#RecentOutsideTemp h=3>) - (11.37 * wind_pow) + (0.3965 * <#RecentOutsideTemp h=3> * wind_pow);// Brackets used to ensure "+" is interpreted as addition not concatenation
} 

For temperatures above 20°C or 68°F Feels Like uses a different way to calculate apparent temperature that it uses at these higher temperatures (this formula only used for 3.6.10 onwards):

var actualVaporPress = <#RecentHumidity h=3>/100) * 6.112* Math.exp((17.62 * <#RecentOutsideTemp h=3>)/(243.12 + <#RecentOutsideTemp h=3>)) / 10.0;   // Not same as at build 3084
		/* uses kilometres per hour for wind speed 	*/
	/*  What Cumulus MX will use to calculate apparent temperature for feels like is changed very slightly */
			if(<#RecentWindSpeed h=3> > 72) <#RecentWindSpeed h=3> =72;
			AT= (1.04 * <#RecentOutsideTemp h=3>) + (2 * actualVaporPress) - (0.1805553 * <#RecentWindSpeed h=3>) - 2.7;

For in-between temperatures it uses a more complicated merge of the two formulas for AT and WC as defined above:

app_temp_mult		= (<#RecentOutsideTemp h=3> - 10) / 10;
wind_chill_mult 	= 1 - app_temp_mult;

FL= AT * app_temp_mult + WC * wind_chill_mult;