Dayfile.txt

This article is about the Daily Summary logging file

Introduction

Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. In all flavours of Cumulus, this file is only updated (with exclusive lock) during the end of meteorological day process. In that process it is also read, if the generation of NOAA reports has been requested.

Both Cumulus 1 and MX have ways to edit dayfile.txt while Cumulus is running. Cumulus 2 does not even allow you to view this log file. If you have a file created by Cumulus 1 (C1), please note that Cumulus 1 is not as fussy as MX about the character used to separate different parts of the date. Also MX uses the locale to decide what character (decimal comma or decimal point) separates integer and decimal parts of numbers, so for MX to read a file created by C1, the MX locale must match the Windows settings.

Please see Correcting Extremes page for information about fixing where the highest/lowest/total recorded for day has been corrupted by rogue values.

The number of fields in the file increases in various versions as shown at the end of this article. C1 can cope with 15 to 45 fields in each line. MX (except for the early releases) assumes 48 fields in each line. If you have some lines in your file that were created by an earlier release of Cumulus, and so have less than 48 fields, you can add the missing derived fields, please see Calculate Missing Values page.

In MX only, the figures contained in the file are used for the historic charts functionality; both in the admin interface, and the example included web page.

In Cumulus 1 only, the figures contained in the file are used for the 'This period' display (and the month/year displays) accessed from the View menu; and to build any graphs based on daily values.

Changes in different releases of Cumulus

Be aware that this article was written for Cumulus 1, amended to also cover MX 3.0.0. Whilst the list of fields has been kept updated for more recent MX versions, other parts of this article may not be applicable to latest MX version, check for edits by developer.

Notes for Legacy Cumulus

• Note for obsolete version 1.9.0 to 1.9.3: There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.
• Note for obsolete version 1.9.3 only: Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order)
• There are no known bugs for dayfile.txt handling in version 1.9.4 builds 1086 to 1101. Build 1099 is the standard stable final release of Cumulus 1 for most weather station types, 1100 and 1101 are for specific weather station types.

When Cumulus is left running

• Cumulus is frequently reading observations from your weather station.
• Cumulus tracks the highs and lows in weather observations by comparing read values against those it has stored in Today.ini, updating that file as required.
• The original Cumulus software reads the contents of dayfile.txt for many of the display and edit menu options that can be selected from the main Cumulus 1 screen.
  • As the file is read each time you select to display a period, or update for a new period, you will see the results of any edit of dayfile.txt, whether with in-built editor or an external editor.

• Cumulus MX beta version 3.0.0 (checked at build 3043) does not provide an editor
• Cumulus MX Version 3.4.5 - Build 3069 onwards provides an editor
• From MX release 3.9.2 - b3097 historic charts are added to admin interface
  • If you use the in-built editor to update dayfile.txt, your change will be reflected in the historic graphs
  • If you edit dayfile.txt outside Cumulus MX, historic graphs will not be updated until Cumulus MX is restarted

When Cumulus processes the end of the (meteorological) day

• A new row is appended to dayfile.txt, the values are prepared from reading "today.ini" file, not all values available in "today.ini" are stored in dayfile.txt.
• Some of this information is also stored in yesterday.ini.
• Back ups of both today.ini, and dayfile.txt, log files in their state after the end of day update are copied to Backup folder.

Options for reading dayfile.txt for other uses

See Daily Summary page for a full discussion.

• Some people take a copy of the local dayfile.txt, and use it locally for other purposes.
• A number of third-party routines that make use of this file (and in some cases, others) are listed at Category:WebTools.
  • Some people use the method described here or Cumulus Toolbox to get a copy of the local file to use on their web server. (If your web server is local, you can simply copy the file to that server, once a day).
  • Search in the Cumulus support forum for examples of third-part JavaScript projects that read the web copy, for example to insert data for one year ago, or to enable extremes for each day in a week to be included in a web page.
  • Dayfile.txt (and other files!) are used as described at the CumulusUtils link.
• Other people load the contents of their dayfile.txt into a database table, allowing SQL enquiries to efficiently display, or summarise, the contents of the file.
  • Cumulus MX provides optional functionality to create a database table, and to automatically insert a new row each time a new line is added to the local log file.

Populating a database table

• The article here describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have been edited for MX, so you will need to use an older version of the script that fits the version of Cumulus 1 you are using.
• Please see MX_Administrative_Interface#Standard_Daily_Summary_Table section for details of how CumulusMX.exe has an optional feature with a standard option to insert a new row into a database table holding columns relating to the dayfile.txt fields. Either (for all early MX releases) ExportMySql.exe, or (for all later MX releases) ExportToMySQL.exe, can update the database table with past rows.

Using a database table

In both cases, your web site can use that database table avoiding any clash of timing with the Cumulus 1 or MX use of the daily summary log.

For examples of some of the third party tools (Cumulus) using the database daily summary table see here.

In my case, I also store the equivalent of what appears on my version of "thismonth.htm" each month in another database table, i.e. I have one database table column for each of the weather derivatives I show on my web page that show this month's values; it is many more derivatives than are shown on the standard web page, but some are initially hidden. Consequentially, when my daily update script detects from the date that it is processing the last day of a month, it then starts another script that reads all the rows in the daily summary table for that month, and stores the highest/lowest/total (as relevant) in my monthly_summary table (nothing to do with the "monthly" table that MX can generate from its standard log file). This monthly summary table allows me to have web pages that compare consecutive months or compare months between years. Just another example of how much you can get from just one (daily summary) log file!

Alternative schemas

Of course you do not need to exactly mimic the log file with the schema in your database table, your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table, or you may wish to add other values from external sensors or other log files.

• 
  • With Cumulus 1, you would need to be a programmer and write your own script to update the database table with your own schema. You might use the importCumulusFile article to start you off.
  • You might also, as I did, want your script to validate what it reads from the daily summary log to ensure only valid numbers and times are stored in your database table, while any invalid inputs are stored as nulls by your script. In my own case, my daily summary table has no solar columns but it does have several additional columns (including the daily increment of chill hours, the cumulative chill hours, the contents of the Weather Diary, the time of the last rain tip, wind bearings as compass characters (e.g. NNW) as well as numerical bearings). When I used Cumulus 1 I wrote a PHP script to find all these additional values, for example it reads the today.ini and month.ini log files as stored in the end of day backup (not the ones being updated for new day in data folder), and the weather diary in log.xml in data folder.

• 
  • MX allows you to specify a different schema in the SQL it generates, but it does not offer that validation feature I just mentioned.
  • MX automatically stores all end of month figures as log files, a feature that Cumulus 1 and 2 lacked, but as yet it does not actually use this extra data, and provides no simple facility to put what is in these files into database tables. There is no end-of-month selection for updates in MX, so you can't easily get as much from dayfile.txt as I do.

When Cumulus is restarted after a break in running

• It reads the daily log and uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see this Cumulus 1 FAQ)
• Thus you must not have another process attempting access to the daily log when Cumulus is re-starting.
• For Cumulus 1, back ups of 8 selected log files including dayfile.txt that are copied to start-up folders in the 'cumulus\backup' folder, the last 8 start-up folders only are retained.
• For Cumulus MX, there are backups of 10 files, the extra ones are the weather diary and Cumulus.ini, that are copied to start-up folders in \CumulusMX\backup\, again there are only 9 kept, unless you back these up somewhere else.

If your weather station includes it own data logger, then Cumulus will read "archived records" from that log, and when it detects the archive log has moved to a new meteorological day, it will run the "end of day" process that moves what is in today.ini into a new line in dayfile.txt.

List of fields in dayfile.txt

The number of fields in a line

The single file dayfile.txt can contain lines created over a long period of time (speciallog.txt is another log file that contains all dates in a single file, as do all the .ini files).

For some people, the file was first created by Cumulus 1 (at various versions over time), and then moved to MX. Such files may have different numbers of fields depending on when each line was either created or last updated. Cumulus does not in normal operation modify earlier lines, but both Cumulus 1 and MX provide editors where it is possible to modify any line, and any line modified will from then onwards have the number of fields defined for the Cumulus version being used..

There is advice on editing the file on Calculate Missing Values page, using the Cumulus in-built editor. Should you want to edit the file outside Cumulus, a "Comma Separated Value" editor has advantage of being able to maintain the differing existing lengths for each line. One example of such a tool in the Microsoft Windows environment is freeware CSV editor software by Sam Francke for Windows. If you do edit the file (within Cumulus or outside), there are Rules to follow.

Variation by Cumulus version

For the original Cumulus software, each line of this file can contain anything from 15 to 45 fields, and having some lines shorter than others does not matter.

For Cumulus MX, some early releases supported only 45 fields, one particular release supported 54 fields, but all recent releases expect exactly 52 fields, and if you have a line with fewer fields then it will have 52 fields when you edit it, and when you save that line.

The dayfile.txt has grown simply because Cumulus's functionality has been extended as time has gone by. To help you, the table below shows fields grouped by the Cumulus version when those fields were added.

For your installed build please see dayfileheader.txt (stored within the folder that contains your Cumulus executable), as that will list which fields are available for you.

Information shown in the table

• The fields are now numbered starting from 1 to fit in with Cumulus MX where when the log file is read, the processing code adds a line number in front of the date field on each line it holds.
  • The Cumulus MX user may not be aware of this happening as it is within the internal workings, where data from the file is transferred to an array, or from the array is written back to the file.
  • By using line numbers, MX is able to identify which line has been deleted or edited ignoring the date (although unlike Cumulus 1, MX does not allow you to change a date)
  • Cumulus 1 does not actually number lines, it counts lines as it reads them, so if there is an error when it reads the file, the original Cumulus will report the line number where it first found an error.
• The original table below was for Cumulus 1 and then field number was starting from zero. So in some forum posts you might see references to old numbering, in others to new numbering. The old numbering from zero had two advantages:
  1. Cumulus 1 stressed that the date field was different to the rest, as it was used as identifier. The date must be a unique identifier, the same date should not be repeated in another line, however Cumulus 1's editor allows you to change that date field.
  2. The remaining fields were all either numerical values, or a time paired with preceding numerical value. Cumulus 1 actually forces this pairing.
  • Numbering starting from zero is consistent with standard indexing used for arrays in programming languages (like JavaScript), so the number shown was the number to quote in any scripts where a line was converted to an array, and you needed to address a single field.
• The alphabetic column identifiers used by many spreadsheets are shown, please see warnings about using spreadsheets for editing earlier on this page
• The type of field is shown, you must not put a sign for an unsigned field, you can not specify a decimal point in an integer field, all time fields must use HH:mm format
• The field description is shown, together with references to where that terminology is explained

List of Fields

Field number	Spreadsheet column	Field type	Description
0	For internal MX purposes, the zero field identifies a field that holds the line number. It is not actually stored as a field in the log file, but precedes any line exchanged via an application programming interface, and therefore is also included in an array representing all the fields in any log file line. If you are processing this log file using a third party (or your own) script, that probably does not place the line number into any array, and your array elements will start at 0 for the field labelled 1 in this table, so putting all field numbers out by 1.
Those fields included below have been in dayfile.txt from the start of Cumulus 1 (Version 1.0, the First release on 27th January 2004).
1	A	8 characters	Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup)
2	B	Unsigned number	Highest wind gust speed
3	C	unsigned integer	Bearing of highest wind gust
4	D	5 characters	Time of highest wind gust
5	E	signed decimal	Minimum temperature
6	F	5 characters	Time of minimum temperature
7	G	signed decimal	Maximum temperature
Consistency Note: In some cases Minimum comes before Maximum, in other cases Maximum is before Mimum
8	H	5 characters	Time of maximum temperature
9	I	Unsigned number	Minimum sea level pressure
10	J	5 characters	Time of minimum pressure
11	K	Unsigned number	Maximum sea level pressure
12	L	5 characters	Time of maximum pressure
13	M	unsigned number	Maximum rainfall rate
14	N	5 characters	Time of maximum rainfall rate
15	O	unsigned number	Total rainfall for the day
Above here represents the minimum length for every line, a count of 15 items
[There is no record of which version added this next field. The Cumulus Support Forum, while it was hosted by Steve Loft, moved to new forum software (phpBBB3) on 2 Jun 2008, and started afresh without retaining any previous content. Therefore all announcements about the content of each build prior to version 1.7.9 were lost. All that can be deduced is that it was between versions 1.2.5 and 1.5.1 as these do not appear in the release history issued by Steve Loft. The first mention of it in the new forum was not until December 2008, but that was not about when it was released. A web tag for this variable was added in Build 978 of 1.9.1 beta, which was obviously long after it was first calculated. Because of that, in "DataEditor.cs" (part of the source code that is compiled into CumulusMX.exe) this addition has "Extended for ???" as a comment]
16	P	signed decimal	Average temperature for the day
(Wind run was added from version 1.8.4)
17	Q	unsigned number	Daily wind run
(The next pair of entries were added from version 1.8.9 build 907 (June 2010) as part of a total redesign of how dayfile.txt was implemented in Cumulus 1)
18	R	unsigned number	Highest Average Wind Speed
19	S	5 characters	Time of Highest Avg. Wind speed
(The two pairs of humidity entries were added in October 2010, a v 1.9.0 beta, the exact build number is now lost)
20	T	unsigned integer	Lowest humidity
21	U	5 characters	Time of lowest humidity
22	V	unsigned integer	Highest humidity
23	W	5 characters	Time of highest humidity
(The next two entries were added from version 1.9.0)
24	X	(not documented)	Total evapotranspiration (Only valid for Davis stations, shows zero otherwise)
25	Y	unsigned	Total hours of sunshine (only valid if sunshine sensor connected)
(The next 16 entries were added from version 1.9.1 May 2011)
26	Z	signed decimal	High Heat index (added to Cumulus in 1.7.11 only as spot value, not stored)
27	AA	5 characters	Time of high heat index
28	AB	Signed decimal	High Apparent temperature
29	AC	5 characters	Time of high apparent temperature
30	AD	signed decimal	Low apparent temperature
31	AE	5 characters	Time of low apparent temperature
32	AF	unsigned number	High hourly rain
33	AG	5 characters	Time of high hourly rain
34	AH)	signed decimal	Greatest wind chill (high wind speed, low temperature) (calculated since version 1.8.3 as spot value, not stored)
35	AI	5 characters	Time of greatest wind chill
(The next two pairs for dew point were added in version 1.9.2 beta build)
36	AJ	signed decimal	High dew point
37	AK	5 characters	Time of high dew point
38	AL	signed decimal	Low dew point
39	AM)	5 characters	Time of low dew point
(The next three entries were added in version 1.9.2 Build 1004)
40	AN	unsigned integer	Today's dominant/average wind direction
41	AO	unsigned decimal	Heating degree days
42	AP	unsigned decimal	Cooling degree days
The next two pairs were added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist)
43	AQ	unsigned decimal	High solar radiation
44	AR	5 characters	Time of high solar radiation
45	AS	unsigned decimal	High UV Index
46	AT	5 characters	Time of high UV Index
The next two pairs were added in version 3.6.0, 2 more derived values and their times
47	AU	signed decimal	High Feels Like temperature
48	AV	5 characters	Time of high feels like temperature
49	AW	signed decimal	Low Feels Like temperature
50	AX	5 characters	Time of low feels like temperature
The next two pairs were added in version 3.6.12 Version 3.6.12 (build 3088) was an emergency release to cure serious problems in previous build 3087. It added the following 4 fields (2 values and their times). The 4 extra fields are left empty in this release, although you can add values and time-stamps using the dayfile editor. From version 3.7.0 the first 2 of these 4 fields are populated, and the last 2 are removed, so I have labelled them as error.
51	AY	signed decimal	High Canadian Humidity Index or Humidex
52	AZ	5 characters	Time of high Humidex
Just confirming that the next 2 fields were included by mistake in an emergency release (3.6.12), and are not included in current nor any other version, so have labelled them as error.
53 (error)	BA	signed decimal	Labelled as Low Humidex, but not used, (appear in 3.6.12, but no other version)
54 (error)	BB	5 characters	Labelled as Time of low Humidex, but not used, (appear in 3.6.12, but no other version)

Example of the file

An extract of a few lines of the dayfile.txt

01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5

02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7

03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9

04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9

05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3

06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3

07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7