Daily Summary

DAILY SUMMARY

Introduction

1. This article updates Annual Data Weather article contributed by David Jamieson (current web site) in March 2011.
2. A working example of the original Annual Weather Data Summary Ver 1.2. November 2012 Script by David Jamieson (PHP variant), is here; note that example uses a different carrier page (see below) from the Weather Blue set of Alternative Templates available from the French only site Météo du Québec and the international site Weather by_You.
3. This article is called Daily Summary as it summarises scripts that various authors have contributed that take as their source daily ststistics output from Cumulus
   • Typically these scripts allow the maximimum, minimum, average, or total of a statistic calculated for an individual day to be seen in a table that shows such values for many days often spanning at least one year.
4. The scripts mentioned below range quite widely:

• Some are direct descendants of David Jamieson's original idea, in that they read the daily summary log file that Cumulus creates, and produce an output that looks like a calendar with months heading the columns and one row per day of the month while the internal content shows how a selected statistic varies from one day to another
  • Others are variants in that the columns might represent years, or you may be able to scroll the columns so although they are months, they don't have to start with January.
  • Some use purely HTML, CSS, and JavaScript (although they use a JavaScript library called jQuery)
  • Others add some PHP Hypertext Preprocessor (PHP) script so that some of the work is done on the web server before the HTML page reaches the browser
  • I have included cross-references to various forum threads that explore variations to the original idea by adding enhancements such as colour shading, day naming (e.g. 'Wednesday'), seasonal summaries, or in some other way offer a different look, but are still generally based on reading the log file.

• However, there are also scripts that are written in PHP and use database queries (typically MySQL) to retrieve the required statistics from a database Table containing these statistics with one table row for each day
  1. The output from these scripts can be a calendar type table, similar to David's concept, that only shows a single weather criterion
  2. The output can be a subset of the database table, in that a number of selected database columns are shown, and there is one row for each day
  3. The output can be for a single criterion, and for each day of the month (or year), figures shown for every year available; this allows instant comparison between tha same day in all years

Finally do be aware that references to external sites either in the Wiki or the Forum may use links that no longer work, the destination site might have closed down, changed its name, or just eliminated its past content in a revamp. If you do want access to a script, or demonstration, that you can no longer find, post in the forum as it might still exist in an archive somewhere, or another Cumulus user may have a copy to share with you.

Update of David Jamieson PHP script by BCJKiwi

Here is a revised set of files (PHP only) ver 1.2. contributed by BCJKiwi on 17 December 2012 that update David's Ver 1.2 (November 2012) fileset to eliminate errors in validation due to non-standard or deprecated code routines. They also include a couple of updates from DAJ version as advised by posts in the Forum.

Errors rectified by this version

• Non-standard attributes 'datacol' and 'datarow' are generated by the original javascript and php routines that will not pass validation against standards (in HTML5 these can be made to validate by changing to 'data-col' and 'data-row').
• In version 1.1 of this tool, jQuery version 1.5 is called, this permitted use of '.attr' object to add/remove the highlight; from jQuery version 1.6 that was deprecated and the '.prop' object is now specified for the same purpose.
• Version 1.0 and 1.1 default to showing a table for viewer's local current calendar year. Remember the dayfile.txt (even if updated each day) only contains records up to the day ending at the last rollover time using the timezone local to the weather station. Timezone differences when combined with rollover time variations can lead to a period of 2 days discrepancy. At New Year a blank table will be displayed during this period by default. For example, 30 or 31 (depending on whether rollover has occured) in December of the last year could be the latest available for a site in USA whilst the original version of the code is showing a blank table of the next year during early morning on 2 January in Austrailia. Even when station and viewer are in same timezone, there will be a blank table until the first rollover on 2 January.
• Version 1.2 defaults to showing the year found in the last row of dayfile.txt.
• The final field on each row is not read correctly, because the row break has been specified wrongly for a Microsoft Windows environment, it should quote fdata.split("\r\n"); i.e. defines that carriage return then line feed (newline) used to split daily observations in dayfile. Because the script presented here only processes a small number of fields from earlier in the row, this error does not affect the output for the published code.

• See more background in forum thread AnnualDataSummary PHP.

Configuration Variables

By default in 'readDayfile.php':

• Assume the dayfile.txt is read from a subfolder called 'data'.
  • Look for the line 'dayfile='/data/dayfile.txt';' around line 15 and change this to point to your dayfile.txt on your webserver.

for PHP version the path should start from the / (root)

• Assume the date format in the dayfile is dd/mm/yy
  • You may need to edit the file if you do not use '/' between the parts of the date (the month is always in the middle for all Cumulus log files), look in file for date_delimiter' - the symbol separating your date format. See setup.
• Assume the data in the dayfile is separated with a comma
  • Look for the line mentioning field_delimiter - the symbol separating each of your fields in the dayfile.txt. For most people this is a comma but (if you use comma to separate integer and decimal parts of real numbers) it could be a semi-colon (;) or other symbol.

Localization / Language

The script has been designed to be easily translated to your language of choice.

As in Instructions below, edit the readDayfile.php

• variable mn is a list of the 12 months of the year, in an abbreviated format. You may change these as necessary, but try to keep it to an abbreviation as there is limited space.
• 'label_items' is a list of all possible data sets to be displayed. Again you can change these to suit if you are prepared to wade through all the script and make changes elsewhere too. However, the format is a little more involved and you should take some care. Each row represents one data set, with 4 columns of settings for that data set; so the default list is an array of four by six.

Example of a row,....

['maxtemp','Max Temp','Maximum Temperature',true]

• The first element, in this case 'maxtemp' -- is a system variable to identify the data set, used for the URL parameter option, used by the Switch coding (so any change here needs to be reflected there), and also tested elsewhere in the code (in the Rainfall and Windgust data sets a particular style is applied to their zero values).

However, if you add a new data set, then your new first element should be a unique identifier.

• Second, Max Temp is the text to be displayed in the button at the top of the table. You may change this to your own language

If you add a new data set, remember to consider how many buttons can be fitted across the top of the table, you may wish to split into multiple rows.

• Third, Maximum Temperature is the text shown at the top of the table to describe the current data set; again you may change this
• Finally, 'true' will display this button at the top; 'false' will hide it. Therefore, if you do not wish to allow users to jump to the 'Rainfall data set' change the 'true' to 'false' in the 'rainfall' element of the variable. (true/false MUST be lowercase)

NOTE: Sfws 12:22, 31 December 2012 (UTC)

Having made any change to the array as DAJ says above, you also need to change what is in a switch section and you will find that is below where DAJ says '// Nothing to edit below here'.

The switch ($whatdata) { ... } part of the coding identifies the columns in dayfile.txt for the values to be displayed. Any change you make in the array, needs a corresponding change within this part of the script. A typical case here reads

case 'mintemp'	: $tablelayout .= $label_items[1][2]; $dayfilecol=4; break;

so to add a new one, copy that line and paste it in again but in this new line replace 'mintemp' with whatever you have added as the first element in your new array row, replace '[1]' with square brackets round the number of your new array row (the first row is number zero) and finally replace '4' with the field number where the parameter you have added is found in dayfile.txt.

Instructions

The instructions in David's original article for PHP version still apply, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant.

• Download the following zip file...AnnualDataSummary_12.zip, the zip contains 3 files:
  1. readDayfile.php - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style
  2. datasummary.php - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) Settings.php and common.php. You can use your own page instead of this file - see carrier page customisation below.
  3. datasummary.css - styling sheet for the table structure that displays selected statistics in a calendar style
• Edit the file 'readDayfile.php' in a good text editor (for example, notepad++) and consider the configuration variables described above that appear in the top section of the script.

• After any necessary editing, copy datasummary.php and readDayfile.php files to your web site into a subfolder, or the root -- your choice
• Copy datasummary.css (for possible tailoring see styling instructions below) to your web site into the same folder

Usage

• Open datasummary.php from your website in a browser

Along the top will be a menu of the (in original version) six data sets available and on the top left of the table is the year currently being shown. Change either the year, or click one of the top buttons to change the data set. Depending on the speed of your internet connection, and your browser, the new web page may take a few seconds to be processed as it involves: re-reading the dayfile.txt file, redrawing the table, and inserting the values.

If you mouse over a particular value, it will be highlighted together with the corresponding day (row heading) and month (column heading).

Opening a specific data set when the page is first loaded

By default (in the original version) the table created will show the available daily values of the maximum temperature for the current local calendar year, however you can start with any data set and any year (assuming you have those values in the dayfile.txt) by adding a 'Query String' to the end of your URL in the browser.

• The parameter 'data' (in the original version) takes mintemp, maxtemp, avgtemp, minmaxt, rainfall, windgust. (You can add further data sets or change the language of these values - see Localization). Note the specified data will be displayed even if it is assigned 'false' and does not appear as a button.
• You can supply both data set and year parameters, one only, or none

examples....,

• myserver.com/datasummary.php?year=2010

This will open the default (maximum temperature) data set but initially showing year 2010

• myserver.com/datasummary.php?data=rainfall&year=2009

This will open the rainfall data set for the year 2009.

Understanding the Basics

The original 'tool' relies on three files for successful operation:

1. the jQuery library routines that allow it to work in any user agent (browser)
2. datasummary.css (the style sheet) and
3. readDayfile.php (the PHP version of the script).

Please note that the table needs a good amount of space to show a full year of data (at least 900 pixels unless you start reducing the font size!)

In addition you need a carrier page to show the data.

• Note that BCJKiwi does not use a web page based on those that are supplied with Cumulus, as mentioned above his provided datasummary.php requires two extra scripts not provided. If you create two empty files with the specified names, or download files with those names from Ken True's web site, then the provided carrier page will load on your site.
• Alternatively don't bother to upload the provided datasummary.php, and instead create your own carrier page (use the same name or call it whatever suits you) following instructions below.

Creating your own Carrier Page

To create your own page, follow another page you are using successfully on your site and modify it by removing any content from <body> section you don't want and including in the <head> section:

• the CSS styling:

(HTML best practice is to include CSS declations as early as possible).....

<link rel="stylesheet" type="text/css" media="screen" href="datasummary.css"  />

• the script library (this example shows version 3.1, you can load whatever is the most recent version suitable for your browser):

This loads a jQuery library from Google servers rather than having the file on your server, (note many versions of jQuery are not supported by old versions of Internet Explorer, use the very old version '1.5' used by DAJ instead of the current version where '3.1' appears if you need to);

<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1/jquery.js" type="text/javascript">

• Next you must load the script to build the table:

In the <body> section, use the code

<?php include('readDayfile.php');?>

(or simply copy everything that was in that file and paste it in this section directly) to include the script instructions that create all the HTML elements required to build and display the table

Styling

The table styling is completely configurable using the included Cascading Style Sheet datasummary.css. By default it is using similar colours to the standard Cumulus website. Below are a few of the key entries to consider when adjusting settings (typically colouring)... Also see support forum thread about colouring with php

In the 'datasummary.css' file:

CSS Element	Description
#table_container .highlight	Used to highlight the mouse position within the table, and also the row and column header of the data cell.
#table_container .smallfont	Adjusting the font size to something smaller when showing both Max & Min temp on one data cell
#table_container .zerovalue	In the Rainfall and Windgust data sets this style is applied to any values of 0. By default, the colour is set to a lighter grey but you could add 'display:none;' to hide zero values completely
#table_container table th	Colouring for the top header (Month names) and left header (day numbers)
#table_container table td,table th	The width of each data cell in the table. Be careful adjusting this as making it too small will stop the data being displayed completely
#table_menu li	The styling for the buttons at the top of the page (those for changing the data set)
#table_container	Set the overall font size and style used in the table, as well as the text colour; table positioning and maximum width

Enhancement of JavaScript script

This section was originated by Sfws to mention script originated 24 Nov 2012 Update of text here 00:53, 18 May 2013 (PDT) Finalised version of script in forum 7:08 am 15 Jan 2014, minor update 8:51 pm 15 Nov 2014

Find my package in alternative Javascript/jQuery solution within the 3rd party tools sub forum of the Cumulus forum.

I originally created the alternative Javascript version in December 2011 by editing the DAJ original (with similar aims to BCJKiwi with his PHP version 1.2) to offer easier customisation plus standards compliance, i.e. addressing all the issues with David's version. If you don't use PHP, this is the script you should install, and you will find various threads on the forum about it in the Web site - General sub-forum as well as posts in the thread containing the source.

It really is much easier to add new data sets if you use this alternative JavaScript version in the forum. That includes more explanation about the original coding, and an alternative simpler way to specify and adjust the columns in a revised version of the array specified for the PHP version above.

The advantages of my script are fully described in a file within the package. They include standards compliance, better integration with standard Cumulus pages, statistical additions to the bottom of each month's column, and easier tailoring to your preferences on wording, language, and text colouring. During 2012-2013, my alternative JavaScript version was rewritten from scratch (losing the original DAJ code), a number of additional features were added, whilst attempting to preserve ease of customisation to suit any requirements you have e.g. on the language and level of diagnostics output. One new feature is ability to scroll sideways, so calendar look can start with any month (any year) but always with 12 columns showing month headings.

One site using in January 2013 a variant based on an early version of my Javascript/jQuery alternative code, with its own look to the page, is found here. Whilst a number of other people have tried later versions of this alternative code, and you can see some commenting in the forum thread, remember this was designed for easy tailoring, so is not an off the shelf package.

At the end of 2014, I rewrote this again, by adding use of PHP script for some of the actions previously done by JavaScript, retaining JavaScript for handling the selection of a change of criterion, and for handling the change of months to be shown (either by scrolling one month at a time or by picking one of the predefined periods on buttons). The resulting script was to prove very useful to me, but with the mix of languages (and the way it was developed) I found it was not easy to understand and hard to improve so I abandoned any idea of a version to share. However, I decided to rewrite that PHP version making it simpler by totally eliminating JavaScript, although given how many PHP versions existed (see below), mine was not shared.

Variants of the original Daily Summary discussed in the support forum

Many of the forum discussions are in the sub-forum: Cumulus > Cumulus 1 > 3rd Party Tools, but there are a few in the sub-forum: Website Development > Web site - General. So you really need to search through both for all available information as it is spread through hundreds of topics. I highlight just the most useful threads below.

Using JavaScript

As well as the alternative JavaScript version above, SFWS produced a few other scripts that were developed from David's original script including Comparing with one year ago, a script that allowed you to compare statistics from yesterday (in Cumulus' yesterdayT.htm style) with those from a year earlier (or any other past date). The forum thread includes feedback and consequent changes.

PHP scripts

Look in the Website Development section of the Cumulus support forum - Web general sub forum:

• Parse dayfile.txt PHP no database - This script by the late Kevin Reed who ran the web site TNETweather.com is discussed in forum, it is a no frills reader written in PHP (change '.txt' into '.php' when you download it) that parses dayfile.txt and produces an output table with headings. The script can be downloaded on Ken True's site from the link quoted, but with the closure of the original web site there is no further information still available.
• 'Annual Data Summary - Coloured Values' - An extensively coloured version of David's original script with some useful discussion on the trade-off between testing values and efficient code here.
• 'Annual data Summary (DAJ script) - show day-of-week' - This is just one of several places on the forum where further ideas are explored by Mark Crossley and beteljuice.
• 'Yet Another Dayfile Reader (PHP)' - Download from here the ultimate PHP parsing dayfile.txt version that has progressed a long way from David's original, it incorporates all the ideas discussed in earlier forum topics, some of the features are: the highlighting makes day of week appear in row headers, you can swap between daily and seasonal views and there is a table summary including averages. See that forum url for links to some of the many sites that have adopted this version, if you do not have a database, but do have PHP, then this is version for you. Mark Crossley has his own offshoot of this script, data summary, that reads a database table, but uses much of the same script so produces a similar summary.
• 'Re: New Script that displays a selected daily statistic for every day' - In #Enhancement of JavaScript script above, Sfws refers to a pure PHP version that reads data from a database table. That version is now available in the linked paragraph, but using a site specific schema.
• 'New Script that displays a selected daily statistic for every day' - Download from here, in the above topic, the newest (at time of writing) way to display daily statistics. This is the "daily pick" script where you select one criterion and all the daily values available are output, for ease of display there is a separate table for each month with HTML <detail> tags to show/hide. Both this and previous scripts were written for a schema that is different to the David Jamieson/Mark Crossley/Cumulus MX standard, but the daily pick script has both schemas available, comment out the one you don't want and remove comments round the one you do want. See further information in posts in the topic and the topic that it follows on from, at MySQL Code Help.
• 'SQL dayfile data Viewer' - This is another (widely adopted) PHP script for reading daily values from a database. It displays a number of key statistics, and you can order the results by the contents of any column. Read the forum topic for more information. You can scroll through the days or set a filter to only show certain days. There is a summary line, but because some columns include units, the calculation of the summary for those columns fails. It is quite simple to edit out the units, so all columns show a summary, but most users adopt the script as published and just accept that some summary figures are replaced by text.

Displaying daily statistics from a database table

Schema

1. The retrieval script needs to have the following defined for it to use (it is suggested that these variables are defined in a separate PHP script that is in a directory that the PHP parser can access, but is not in the part of the web server that is visible to a client):
   • The server host name, or w.x.y.z format Internet Protocol (=IP) address, running your MySQL database
   • The user name used to log-in to your database server
   • The password used to log-in to your database server
     • (It is good practice to have separate authentication for users with privileges of read only access, and for users with privileges of read and update access, alternatively use an additional key variable as David does in his update script described below)
   • The name of the (MySQL or whatever) database with the table
   • The name for the daily summary table
2. The retrieval script needs to know the names of the columns it is to read from the database.
   • Database designers talk about the schema meaning the specification that describes the design of the table.
   • A schema lists the column names together with what type and size of variables they hold.
   • For the daily summary database table, the columns can copy the fields in the daily summary log file, and this is what David Jamieson does in his approach a PHP script to read the contents of the dayfile.txt where the columns are generated in the same order as the corresponding fields in the log file, so David's script uses the field numbers to identify the columns by Position (rather than by Column Name).
   • Remember that while the fields in the log appear in the order that statistic was added during Cumulus 1 development, in theory the columns can appear in any order (if all your scripts identify the columns by name rather than position) so columns associated with rain (these are separated in dayfile.txt fields 12, 13, 14, 31, and 32) can appear together, as can those for wind (these are dispersed in dayfile.txt fields 1, 2, 3, 16, 17, 18, and 39).
3. If you have the skills to write your own script, then you might want to consider a different schema, daily information is in three log files for Cumulus 1, "dayfile.txt" should have the statistics for all completed days (that excludes today), "today.ini" has values for today-so-far, and "log.xml" is the Weather Diary where snow records (and general weather text) can be manually recorded. A schema that combines information extracted from those three files to show everything Cumulus has for a day is available in this forum post.

Loading and Updating the database table

Obviously if you are using a script that reads statistics from a table, you also need to be running a script that ensures the table is updated at the end of each day, and therefore always up to date.

• Functionality is included in Cumulus MX to do this update, it uses the schema devised by Mark Crossley by default, but Cumulus MX can also handle your own schema; for full details of the functionality see Cumulus MX Announcements - PLEASE READ THESE!.
• Cumulus 1 users need to use a script to create a database table, populate and update it from the daily summary log file. The most widely used script is an update by Mark Crossley of the script mentioned above by David Jamieson; and it is described in this Wiki article.
  • It is however worth also reading the forum post a PHP script to read the contents of the dayfile.txt as that explains the date delimiter, how a Comma Separated Variable file may use other punctuation to separate the fields, how to specify the location of your dayfile.txt log to the script and the reason the key variable is used. The thread contains several insights into how Cumulus was developed and it is a definite must to read Steve's posts in that thread if you are developing your own scripts.

Some example Scripts

Look in the Website Development section of the Cumulus support forum - Web general sub forum:

• SQL dayfile data Viewer - This script by Kocher shows selected statistics, you can scroll up and down through all available days and for those days (or all days) see some summary statistics at the foot of each column.
• MySQL code help - This shows a snippet of a PHP script by SFWS for creating a table that for a particular criterion shows the value with all the years as column headers and all the days of a year (1 Jan to 31 Dec) as rows. The provided script uses standard HTML to show/hide each month.
• Top 10 Records Page- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. ). The latest source php can be seen using a query-string on Mark Crossley's web site and there have been several updates since the forum thread was created.
• What to do with data from MySQL - This posting lists a number of web pages on PaulMy's web site where he presents statistics from the daily summary table of his database in different formats.
• This day last year page - This post has a query in a php file (as you save the download change '.txt' to '.php') that will extract from the daily summary table in a database, values for one year ago, two years ago to feed a web page that displays all the summary statistics for today and the corrsponding ones for earlier years.

There are various scripts out there that output graphs from the values read, I have not identifed a particular one to reference here yet. If you find a site dislaying a graph you would like to copy on your own site, you might be able to see the PHP source that is responsible, but as I have commented elsewhere most authors do not make it easy to see their code. Of course one of the reasons for using PHP script is that processing is done quickly on the web server and only the resulting HTML is passed onto the client browser and this does protect interlectual property and adds security by not showing the database access code.

Other jQuery approaches

If you extract all the values off the daily summary table with a simple "SELECT * FROM name_of_your_table" query and store those values in XML format using a PHP script called 'read_dayfile_historic.php', then you can use a jQuery library routine to produce a table of values using JavaScript:

<script src="jquery.jqGrid.min.js" type="text/javascript"></script>
	<script type="text/javascript">
	jQuery(document).ready(function(){ 
	jQuery("#list").jqGrid({
		url:'read_dayfile_historic.php',
		datatype: 'xml',
		height: 460,
		width: 970,
		mtype: 'GET',
		colNames:['Date','Highest<br/>Gust<br/>(mph)','Wind<br/>Bearing','Time of<br/>Gust','Min. Temp<br/>(°C)','Time of<br/>Min.','Max. Temp<br/>(°C)','Time of<br/>Max.','Avg. Temp<br/>(°C)','Lowest<br/>pressure<br/>(mb)','Time of<br/>low','Highest<br/>pressure<br/>(mb)','Time of<br/>high','Highest<br/>Rain rate<br/>(mm/hr)','Time of<br/>High Rate','Total Rain<br/>fall (mm)','Tot.Wind<br/>run'],
		colModel :[ 
		{name:'UKLogDate', index:'LogDate', width:65, label: 'Date', sortable: true}, 
		{name:'HighestWindGust', index:'HighWindGust', width:50, align:'right'}, 
		{name:'HWindGBear', index:'HWindGBear', width:50, align:'right'}, 
		{name:'THWindG', index:'THWindG', width:55, align:'right', sortable: false}, 
		{name:'MinTemp', index:'MinTemp', width:55, align:'right'}, 
		{name:'TMinTemp', index:'TMinTemp', width:55, align:'right', sortable: false},
		{name:'MaxTemp', index:'MaxTemp', width:60, align:'right'},
		{name:'TMaxTemp', index:'TMaxTemp', width:55, align:'right', sortable: false},
		{name:'AvgTemp', index:'AvgTemp', width:57, align:'right'},
		{name:'MinPress', index:'MinPress', width:60, align:'right'},
		{name:'TMinPress', index:'TMinPress', width:50, align:'right', sortable: false},
		{name:'MaxPress', index:'MaxPress', width:60, align:'right'},
		{name:'TMaxPress', index:'TMaxPress', width:50, align:'right', sortable: false},
		{name:'MaxRainRate', index:'MaxRainRate', width:55, align:'right'}, 
		{name:'TMaxRR', index:'TMaxRR', width:60,  align:'right', sortable: false}, 
		{name:'TotRainFall', index:'TotRainFall', width:55, align:'right'}, 
		{name:'TotWindRun', index:'TotWindRun', width:55, align:'right'}],
		pager:'#pager',
		rowNum:19,
		rowList:[10,20,30],
		viewrecords: true,
		sortname: "LogDate",
		sortorder: "desc", 
		caption: 'Daily Statistics'
	});  
	}); 
	</script>

The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them.