Dayfile.txt

From Cumulus Wiki
Jump to navigationJump to search

Cumulus maintains a daily log file that holds the highs and lows of each day, as well as a few other nuggets of information. The figures contained in the file are used for the 'This period' display accessed from the View menu and to build any graphs based on daily values.


How Cumulus uses the daily log

  • Cumulus is frequently reading observations from your weather station.
  • It tracks the highs and lows in those observations by comparing read values against those it has stored in Today.ini, updating that file as required.
  • (It also updates Alltime.ini, Monthlyalltime.ini, Year.ini, and Month.ini when appropriate.

See the Wiki pages for these files or the Cumulus Help for more information, including the details of where the previous values are logged for updates to these highs and lows).

  • When Cumulus processes the end of the (meteorological) day, the highs, lows and other information in 'today.ini' are used to append a new line into dayfile.txt, then a new 'today.ini' is created initialised with the observations just read from the weather station.
  • When Cumulus is restarted after a break in running,
    1. it uses the rainfall totals for each day stored in the daily summary log to calculate the rainfall for this month, and this year/season (see FAQ)
    2. depending on the make of weather station:
      • read observations from the station's log into the Cumulus monthly log, updating today.ini as it processes the station log;
      • otherwise the observations in the station's current measurements holder at the Cumulus logging interval are what gets written into the monthly log, and update today.ini.
  • Back ups of all the files mentioned here are copied to the 'cumulus\backup\daily' folder.
  • There are no updates to dayfile.txt at any other times, but the contents of the file are read and processed for many of the display and edit menu options when selected from the main Cumulus 1 screen.

How you can use the daily log

  • If you want to run scripts that use the daily log, it is best if you take a copy first, I take a copy that is put onto my web server by using the Daily box in the bottom left of the Sites/Options frame within the Internet options screen from the Configuration menu to safely take a copy of 'dayfile.txt' after it is updated. See Cumulus Help for information on using this feature, I add a redirection ">daily_batch.log" in the parameter box alongside so that any output from my "T:\Cumulus\daily_batch_all.cmd" in the main box is sent to a log file overwritten in each run; this enables me to see the reason for any failure. My Cumulus 1 batch command also runs PHP to create SQL to update a database table version of the log, but my database table version also contains some additional information from the daily backup 'today.ini' log.
  • The system routines that Cumulus uses to access dayfile.txt require exclusive use of that file, so if you have any other process trying to access that file at end of the (meteorological) day, or when a relevant option is selected from View or Edit menus, one or other may fail.
  • You can find examples of code that uses the daily log elsewhere in this Wiki and on the support forum.
  • Cumulus MX includes the ability to generate SQL to update a database table version of the log. Using that database table avoids any clash with the Cumulus MX update of the daily summary log.

Viewing or Editing the daily summary log within Cumulus

This section applies to Cumulus 1.x.y only - Cumulus MX v3.0.0 (checked at build 3043) does not provide an editor

The last command in Edit menu is dayfile.txt. This is how you view the dayfile.txt from within Cumulus. This is a text editor, so you can type new values over those currently displayed, insert and delete rows, and it works best when at full screen. Click the Help button for detailed instructions. Cumulus Help is comprehensive.

If a particular day does not exist as a row on the daily summary log, then 'create missing' can search the observations in the relevant monthly log, and calculate approximate highs, lows and totals to insert as an exta row in the daily summary log. These are approximate because the actual highs and lows for that day are quite likely to have occured at moments inbetween those that were logged.

Editing daily summary in Cumulus

If there is an error in dayfile.txt, then it is most likely to be found when you are viewing its data on one of the screens for editing the monthly, annual or all-time extremes. Cumulus will illuminate its Error light if it finds an error in such cases and tell you the line/row number of the first found error, together with some details of the error it found. For example, if a row is blank, a row is duplicated, a field is corrupted, a field does not have an acceptable value, or a field is missing so subsequent fields are to the left of where they should be.

You can use this editor as follows:

  • use insert key to add one or more missing rows (complete days) manually,
  • correct individual values by overtyping,
  • use delete key to remove an entire day (e.g. if you get a 'duplicate' error message),
  • use Create missing button to insert missing rows (complete days) by reading from Monthly_log_files and automatically calculating the best approximations for each field for those missing days.

If you do have a 'duplicate' error, you need to decide which row to delete, and whether to copy any values from that row into the row you are keeping to ensure the correct extremes are retained.

As an alternative to manual line insert, you may wish to use a procedure for importing, and processing, pre-Cumulus observations into Monthly_log_files. Once there is data for required days in monthly logs, Create missing can insert the new rows for those days previously missing in dayfile.txt.

For Create missing a list of inserted records is produced in dayfileeditlog.txt. If just some fields are wrong in a particular row (meteorological day) on day file, then there is a workaround as at all current versions (up to 1.9.4) you can only use 'Create missing' to read from the Monthly_log_files if the whole day (a row starting with a date) is missing in dayfile.txt.

Read this if you are not using the latest release of Cumulus

Note for obsolete version 1.9.0 to 1.9.3: There is a bug in these versions in that 'Create missing' inserts 'heating and cooling degree day' values the wrong way round.

Note for obsolete version 1.9.3 only: Create missing might in some cases be affected by a bug in 1.9.3 that can cause incorrect date order for records (dayfile.txt uses dd/mm/yy or dd-mm-yy and all records should be in ascending chronological order)

There are no known bugs in version 1.9.4 builds 1086 to 1100. Build 1099 is the standard release as this section was updated.

Dealing with rogue measurements

Cumulus provides via Configuration menu Calibration screen the ability to screen out spikes (magnitude of differences between one value read and next value read) in data picked up from your weather station. See Cumulus help screen if you decide to use that to cope with some spikes.

If you discover a rogue measurement (perhaps the wind affected your tipping bucket rain gauge or your weather station just reported a corrupted value), on the day it occured, see today.ini or a FAQ for further advice. In general, you need to stop Cumulus, edit the monthly log row containing the dodgy values, edit 'today.ini' and possibly other '.ini' files, looking up the logs indicated below that show the updates with previous high or low.

If the rogue measurement is discovered some days after it occured, then in many cases it will have affected your highs and lows for the current month, month-by-month, current year, and/or all-time. As your first step you should update the appropriate field in the row for the affected date in dayfile.txt. Once dayfile.txt is correct the Cumulus editors will allow you to:

  • update the highs and lows in Alltime.ini by choosing all time records from the Edit menu. See Alltimelog.txt for current and previous values to help you know what rogue value to hunt for and know what the high/low value was before the rogue affected it.
  • update the highs and lows in year.ini by choosing This year's records from the Edit menu.
  • update the highs and lows in month.ini by choosing This month's records from the Edit menu. See Diags for current and previous values of high or low in the current month or the immediate preceding month if the rogue was recorded less than 10 days ago.
  • update the highs and lows month-by-month in monthlyalltime.ini by choosing Monthly records from the Edit menu. Click the Help button for specific instructions on using Reset and the two Copy column header buttons in this Monthly Records (Highs and Lows) Editor to action all rows.
  • Note in each of above 4 editing screens you can:
    1. see the currently stored extremes, and optionally Reset (row by row) to pre-editing value and timestamp.
    2. load the dayfile.txt to view extremes derived from those figures (after your correction of the rogue values) and
    3. optionally Copy (row by row) the logged values (and associated date/time information) into the relevant .ini file.
    4. click the Help button for detailed instructions on using The Records (Highs and Lows) Editors.
    5. store your revised figures by clicking OK (or abandon all your edits by clicking Cancel).
 (Each of these screens is a text editor, and works best when at full screen).
  • create the relevant monthly and/or annual NOAA style report by choosing NOAA Monthly Report or NOAA Annual Report from the View menu, then select the required period using the selectors. Click the Update Display button to see various statistics (including mean temperature) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press Save button to store the new or amended report.

Ideally, you will subsequently try to edit the rogue data for the particular time it was logged; see Monthly_log_files#Correcting_any_logged_data_problems, but correcting the daily summary in dayfile.txt must always be the priority.

Using the daily summary log on your web-site

If you upload the file to your web site, then see AnnualDataSummary for information about one of the ways to show values from this file. Search the Cumulus support forum to see ways that others extract information from dayfile.txt to display on their web page a set of fields similar to those shown for 'Yesterday' web page for other dates in the past. Other people use dayfile.txt for updating databases stored on their web sites so that SQL can be used to display all sorts of statistics, again postings on the forum discuss this.

Viewing summary figures for a month or period

To view a summary of dayfile.txt for a month, calendar year or selected period, use This month (choose any month, default is month from your computer system date), This year (choose any year, default is year from your computer system date), or This period (choose any start and end dates, default is yesterday calculated from your computer system date), within the View menu.

  • Remember the daily summary log has its records based on rollover to rollover days.
  • In all cases they exclude the today details that are not stored on dayfile.txt until the end of day rollover.
  • If you use 9am or 10am rollover, and choose View This period between midnight and your 9am/10am rollover any day your latest meteorological day is the yesterday in terms of your computer system date that 'This period' tries to display as its default day, and the display will initially appear blank.
  • If you use 9am or 10am rollover, and choose View This month before your 9am/10am rollover on the first day of a new calendar month your latest meteorological month is different to your computer system month 'This month' tries to display as its default month, and the display will initially appear blank.

Most of the displayed results are for observations in the daily summary log, but a few parameters are not in that log and are derived from the monthly logs (e.g. average wind speed) or the weather diary (e.g. count of days with snow lying).

  • On the screen displayed after selecting This month, you can change the month and year required using the options at bottom left, click Update Display and the revised summary will be calculated.
  • On the screen displayed after selecting This year, you can change the year required using the options at bottom left, click Update Display and the revised summary will be calculated.
  • On the screen displayed after selecting This period, you can change the start date and end date then click Update Display to get the equivalent calculations displayed for part of a month or any other period.


Note differences between observation reports on View screens and those available as web tags.

  • Date and time stamps:
    • The day number shown on screen is the meteorological day (changing at rollover and that may be at midnight or 9am/10am) as that date appears in dayfile.txt;
    • A time-stamp (with time and date) given in a web tag quotes a calendar date (always changing at midnight).
  • Reported statistics example:
    • The screen shows total number of dry or wet days in the month;
    • The web tags report longest dry or wet period in the month.


List of fields in the file

The appropriate list of fields for your installed build is stored as dayfileheader.txt within the folder that contains your Cumulus executable.

Listed below is the field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript, plus spreadsheet column identifier) with description and cross-references to further explanation:

  • 00 (A): Date as 2 figure day [separator] 2 figure month [separator] 2 figure year - the separator is that set in the windows system short date format (see setup)
  • 01 (B): Highest wind gust
  • 02 (C): Bearing of highest wind gust (integer)
  • 03 (D): Time of highest wind gust
  • 04 (E): Minimum temperature
  • 05 (F): Time of minimum temperature
  • 06 (G): Maximum temperature
  • 07 (H): Time of maximum temperature
  • 08 (I): Minimum sea level pressure
  • 09 (J): Time of minimum pressure
  • 10 (K): Maximum sea level pressure
  • 11 (L): Time of maximum pressure
  • 12 (M): Maximum rainfall rate
  • 13 (N): Time of maximum rainfall rate
  • 14 (O): Total rainfall for the day

(This point represents the minimum length of all records)

(The last two entries were added from v 1.8.9 build 907)

(Those two entries were added from v 1.9.0)

  • 19 (T): Lowest humidity (integer)
  • 20 (U): Time of lowest humidity
  • 21 (V): Highest humidity (integer)
  • 22 (W): Time of highest humidity
  • 23 (X): Total evapotranspiration (Only valid for Davis stations, shows zero otherwise)
  • 24 (Y): Total hours of sunshine (only valid if sunshine sensor connnected)
  • 25 (Z): High Heat index
  • 26 (AA): Time of high heat index
  • 27 (AB): High Apparent temperature
  • 28 (AC): Time of high apparent temperature
  • 29 (AD): Low apparent temperature
  • 30 (AE): Time of low apparent temperature
  • 31 (AF): High hourly rain
  • 32 (AG): Time of high hourly rain
  • 33 (AH): Low wind chill
  • 34 (AI): Time of low wind chill

(The last 16 entries were added in version 1.9.1)

  • 35 (AJ): High dew point
  • 36 (AK): Time of high dew point
  • 37 (AL): Low dew point
  • 38 (AM): Time of low dew point

(Last 4 entries added in version 1.9.2 beta build)

(These last three entries were added from 1.9.2 Build 1004)

Finally, added in version 1.9.3 build 1036 (these only show valid values if appropriate sensors exist)

  • 42 (AQ): High solar radiation
  • 43 (AR): Time of high solar radiation
  • 44 (AS): High UV Index
  • 45 (AT): Time of high UV Index

(Cumulus MX uses all fields as listed above)

Example of the file

An extract of a few lines of the dayfile.txt 

01/08/11,19.3,61,10:22,12.5,06:58,23.8,14:49,1014.26,20:46,1018.83,09:28,0.0,00:00,0.0,17.8,21.6,4.6,10:44,36,14:14,86,01:56,3.56,8.9,23.8,14:49,23.1,14:50,12.3,06:59,0.0,00:00,12.5,06:58,11.3,00:16,6.9,14:34,354,2.0,1.5

02/08/11,16.1,20,16:55,14.7,06:45,24.2,13:54,1013.79,19:13,1015.65,11:14,0.0,00:00,0.0,18.9,13.7,8.0,15:55,42,20:42,85,06:50,2.79,4.9,24.2,13:54,24.3,13:55,15.1,06:40,0.0,00:00,14.7,06:45,14.8,11:59,7.0,21:09,57,1.0,1.7

03/08/11,14.5,36,17:23,14.9,05:50,24.6,14:46,1012.70,18:44,1015.99,08:34,0.0,00:00,0.0,19.4,17.2,4.8,16:04,50,14:38,79,07:04,3.05,5.8,24.6,14:46,25.4,14:47,15.0,05:50,0.0,00:00,14.9,05:50,14.2,20:01,8.9,00:16,32,0.8,1.9

04/08/11,17.7,16,15:43,14.1,06:20,25.3,15:06,1013.08,18:42,1015.31,08:28,0.0,00:00,0.0,20.2,19.4,8.1,14:12,52,18:20,92,06:55,3.30,9.1,25.3,15:06,26.8,14:55,14.9,06:20,0.0,00:00,14.1,06:20,15.8,14:55,12.5,06:25,36,1.0,2.9

05/08/11,16.1,32,12:52,14.2,06:12,22.2,14:07,1013.89,00:01,1016.36,09:43,0.0,00:00,0.0,18.6,21.6,5.2,13:00,62,15:57,87,06:11,3.30,8.4,22.2,14:07,23.5,14:10,14.8,07:19,0.0,00:00,14.2,06:12,15.4,10:33,12.0,06:03,34,0.9,1.3

06/08/11,16.1,309,11:15,14.3,05:29,22.4,17:12,1014.46,20:02,1016.97,10:38,0.0,00:00,0.0,18.4,19.2,5.5,16:21,55,13:33,92,05:20,2.79,7.9,22.4,17:12,23.3,18:17,15.1,06:09,0.0,00:00,14.3,05:29,14.2,18:12,10.9,10:38,32,1.1,1.3

07/08/11,17.7,342,13:24,12.9,05:47,24.1,14:53,1013.92,19:49,1016.43,09:36,0.0,00:00,0.0,18.4,19.1,6.3,14:06,48,12:45,89,05:36,3.30,9.0,24.1,14:53,24.6,15:48,13.3,05:47,0.0,00:00,12.9,05:47,14.6,15:52,10.7,11:33,11,1.6,1.7
Retrieved from "https://www.cumuluswiki.org/index.php?title=Dayfile.txt&oldid=4225"