Daily Summary: Difference between revisions
m (→Introduction: 2 spelling corrections) |
(Updated links, changed sequence of material on page) |
||
(12 intermediate revisions by 2 users not shown) | |||
Line 1: | Line 1: | ||
DAILY SUMMARY |
|||
[[Category:WebTools]] |
[[Category:WebTools]][[Category:User Contributions]] |
||
=Available options= |
|||
== Introduction == |
|||
# This article updates [[AnnualDataSummary|Annual Data Weather article]] contributed by David Jamieson ([http://kippfordweather.uk current web site]) in March 2011. |
|||
# A working example of the original Annual Weather Data Summary Ver 1.2. November 2012 Script by David Jamieson (PHP variant), is [http://www.fordingbridgeweather.co.uk/datasummary.php here]; note that example uses a different carrier page (see below) from the Weather Blue set of Alternative Templates available from the French only site [http://meteoduquebec.com/ Météo du Québec] and the international site [http://weatherbyyou.com/demo.php <strong>Weather</strong> <small>by</small><sub>You</sub>]. |
|||
# This article is called '''Daily Summary''' as it summarises scripts that various authors have contributed that take as their source daily statistics output from Cumulus |
|||
#* Typically these scripts allow the maximimum, minimum, average, or total of a statistic calculated for an individual day to be seen in a table that shows such values for many days often spanning at least one year. |
|||
# The scripts mentioned below range quite widely: |
|||
* Some are direct descendants of David Jamieson's original idea, in that they read the ''[[dayfile.txt|daily summary]]'' log file that Cumulus creates, and produce an output that looks like a calendar with months heading the columns and one row per day of the month while the internal content shows how a selected statistic varies from one day to another |
|||
** Others are variants in that the columns might represent years, or you may be able to scroll the columns so although they are months, they don't have to start with January. |
|||
** Some use purely HTML, CSS, and JavaScript (although they use a JavaScript library called '''[http://jquery.com/download/ jQuery]''') |
|||
** Others add some PHP Hypertext Preprocessor ('''[http://php.net/ PHP]''') script so that some of the work is done on the web server before the HTML page reaches the browser |
|||
**I have included cross-references to various forum threads that explore variations to the original idea by adding enhancements such as colour shading, day naming (e.g. 'Wednesday'), seasonal summaries, or in some other way offer a different look, but are still generally based on reading the log file. |
|||
This page describes routines that either read [[Dayfile.txt]] log file, or run queries against a database table with one row per [[Meteorological day|day]]. The various scripts then display either a selection of the rows, or a selection of the derivatives in a more friendly way that just browsing the file or table. In general, these scripts display some summary information, such as highest/lowest in month/season/year, and some scripts use coloured fonts to group values. |
|||
There is discussion about many of the available scripts in the Cumulus Support forum section called "Website Development", mostly (but not exclusively) in [https://cumulus.hosiene.co.uk/viewforum.php?f=18 3rd Party Tools sub-forum], and/or [https://cumulus.hosiene.co.uk/viewforum.php?f=14 Web site - general subforum]. Where there is information in this Wiki, links are given in summary below. |
|||
* However, there are also scripts that are written in PHP and either read the dayfile.txt or use database queries (typically '''[https://dev.mysql.com/doc/ MySQL]''') to retrieve the required statistics from a database Table containing these statistics with one table row for each day |
|||
*# The output from these scripts can be a calendar type table, similar to David's concept, that only shows a single weather criterion |
|||
*# The output can be a subset of the database table, in that a number of selected database columns are shown, and there is one row for each day |
|||
*# The output can be for a single criterion, and for each day of the month (or year), figures shown for every year available; this allows instant comparison between the same day in all years |
|||
The scripts use various computer languages: |
|||
* C# - see [[Yadr]], (a novice might prefer this first choice as it is part of [[:Category:CumulusUtils|a package]], so there is no code to edit) |
|||
* jQuery - a number of scripts are available, if you make use of [http://jquery.com/download/ jQuery] including: |
|||
** see the [[AnnualDataSummary|Annual Data Summary page]] contributed by David Jamieson ([http://kippfordweather.uk current web site]) in March 2011, (be aware that this requires some user configuration (see towards end of this Wiki page), so some understanding of JavaScript helps, and you need to load jQuery). |
|||
** see [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=8690 Alternative Javascript/jQuery solution] within the '''3rd party tools''' sub forum of the Cumulus forum. (The advantages are discussed in file within December 2011 package, but essentially this has similar output to previous, but is far easier to customise) |
|||
** see [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=8631&hilit=one+year+ago#p71709 Comparing with one year ago] written in style of the yesterday web page provided with legacy Cumulus 1. |
|||
* PHP - a number of scripts are available (if you use [[Your Own Server]] then you need to install PHP), including: |
|||
*# Be aware, some of these scripts only work with the syntax available in the PHP version that was current when the script was written |
|||
*# Be aware, scripts may use "include ..." or "require ..." to bring in variables (such as those for accessing the database) or functions (to do standard actions) or templates (shared headers, navigation menus and footers) or text (any other common content) that is used with a number of pages on author's site. If you don't comment out those references to external scripts, or replace the references with your equivalents, the PHP parser may fail to produce a HTML page and nothing will be available to the browser. Thus you do need a simple understanding of PHP, a PHP book, or the ability to look up PHP instructions on the web if you are to be successful with tailoring these scripts to your site. |
|||
*# If you find a site displaying a graph or table you would like to copy on your own site, you might be able to see the PHP source that is responsible, but (see [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16425&p=126065#p126065 here]) most authors do not make it easy to see their PHP code. Of course one of the reasons for using PHP script is that processing is done quickly on the web server and only the resulting HTML is passed onto the client browser. This inability to see the PHP code that generates the web page does protect intellectual property and adds security against hacking by not showing the database access code. |
|||
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=11397&start=405#p91488 'Yet Another Dayfile Reader (PHP)'] - Download from here the ultimate PHP parsing dayfile.txt version that has progressed a long way from David's original, it incorporates all the ideas discussed in earlier forum topics, some of the features are: the highlighting makes day of week appear in row headers, you can swap between daily and seasonal views and there is a table summary including averages. See that forum url for links to some of the many sites that have adopted this version, if you do not have a database, but do have PHP, then this is version for you. Mark Crossley has his own offshoot of this script, [https://weather.wilmslowastro.com/datasummary.php?view=getorfmiland data summary], that reads a database table, but uses much of the same script so produces a similar summary. |
|||
** [http://saratoga-weather.org/legacy-scripts/dayfile.txt Source: Parse dayfile.txt PHP no database] - This script by the late Kevin Reed who ran the web site TNETweather.com, it is a no frills reader written in PHP (change '.txt' into '.php' when you download it) that parses dayfile.txt and produces an output table with headings. The script can be downloaded on Ken True's site from the link quoted, but see the forum reference [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=2419#p20155 Forum: Parse dayfile.txt PHP no database] for the only further information still available. |
|||
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=5319 'Annual Data Summary - Coloured Values'] - An extensively coloured version of David's original script modified by Mark Crossley with some useful discussion on the trade-off between testing values and efficient code here. The source link in that thread no longer works, but (at time of typing this) the source may be found [http://www.komokaweather.com/weather/datasummary-colour2.php?view=sce here on Paul's site]. |
|||
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=11133 'Annual data Summary (DAJ script) - show day-of-week'] - This is just one of several places on the forum where further ideas are explored by Mark Crossley and beteljuice. |
|||
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16891#p130117 'Re: New Script that displays a selected daily statistic for every day'] - In [[#Enhancement of JavaScript script]] above, [[User:Sfws|Sfws]] refers to a pure PHP version that reads data from a database table. That version is now available in the linked paragraph, but using a site specific schema. |
|||
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16891#p129796 'New Script that displays a selected daily statistic for every day'] - Download from here, in the above topic, the newest (at time of writing) way to display daily statistics. This is the "daily pick" script where you select one criterion and all the daily values available are output, for ease of display there is a separate table for each month with HTML <detail> tags to show/hide. Both this and previous scripts were written for a schema that is different to the David Jamieson/Mark Crossley/Cumulus MX standard, but the daily pick script has both schemas available, comment out the one you don't want and remove comments round the one you do want. See further information in posts in the topic and the topic that it follows on from, at [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16880 MySQL Code Help]. |
|||
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=15983 'SQL dayfile data Viewer'] - This is another (widely adopted) PHP script for reading daily values from a database. It displays a number of key statistics, and you can order the results by the contents of any column. Read the forum topic for more information. You can scroll through the days or set a filter to only show certain days. There is a summary line, but because some columns include units, the calculation of the summary for those columns fails. It is quite simple to edit out the units, so all columns show a summary, but most users adopt the script as published and just accept that some summary figures are replaced by text. |
|||
**[https://cumulus.hosiene.co.uk/viewtopic.php?p=87081#p87081 Top 10 Records Page]- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. lowest daily temperature). |
|||
**[https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16534#p126964 What to do with data from MySQL] - This posting lists a number of web pages on PaulMy's website where he presents statistics from the daily summary table of his database in different formats. |
|||
** [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=8631 Comparing with one year ago] - This post [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=7834#p66013 as requested here] has a query in a php file (as you save the download change '.txt' to '.php') that will extract from the daily summary table in a database, values for one year ago, to display alongside for example statistics for yesterday. An equivalent script using jQuery to read dayfile.txt directly was listed earlier. If you have some programming skills, you can vary either of these scripts to for example display statistics for each day in the last week (or month). I have written scripts that display the statistics for last week using a combination of this daily table in database approach and Cumulus recent tags. I have another script that uses the daily table in the database to calculate the equivalent of what Cumulus 1 shows in View menu, this month selection, for any past month and adds the equivalent of what Cumulus this month web tags show and the resulting statistics are shown for various selectable mixtures of past months. |
|||
== Graphical output from dayfile.txt == |
|||
Finally do be aware that references to external sites either in the Wiki or the Forum may use links that no longer work, the destination site might have closed down, changed its name, or just eliminated its past content in a revamp. If you do want access to a script, or demonstration, that you can no longer find, post in the forum as it might still exist in an archive somewhere, or another Cumulus user may have a copy to share with you. |
|||
== Update of David Jamieson PHP script by BCJKiwi == |
|||
Here is a revised set of files (PHP only) ver 1.2. contributed by BCJKiwi on 17 December 2012 that update David's Ver 1.2 (November 2012) fileset to eliminate errors in validation due to non-standard or deprecated code routines. They also include a couple of updates from DAJ version as advised by posts in the Forum. |
|||
There are '''various scripts out there that output graphs''' from the values in recent history web tags, or values read off either a daily table in a database or dayfile.txt directly. There are two to highlight here: |
|||
=== Errors rectified by this version === |
|||
* An alternative to the Cumulus 1 or Cumulus MX trend web page is described in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=12321 this topic] this can use all the sources mentioned. |
|||
*Non-standard attributes 'datacol' and 'datarow' are generated by the original javascript and php routines that will not pass validation against standards (in HTML5 these can be made to validate by changing to 'data-col' and 'data-row'). |
|||
* Highcharts (that cannot be used on a business website) are exploited by Mark Crossley in [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=8672 this topic]; it is a very long topic, and discusses various issues to consider with using such data, as well as including the script. Cumulus MX now implements this both in its interface pages and its default web pages. |
|||
*In version 1.1 of this tool, jQuery version 1.5 is called, this permitted use of '.attr' object to add/remove the highlight; from jQuery version 1.6 that was deprecated and the '.prop' object is now specified for the same purpose. |
|||
*Version 1.0 and 1.1 default to showing a table for viewer's local current calendar year. Remember the dayfile.txt (even if updated each day) only contains records up to the day ending at the last rollover time using the timezone local to the weather station. Timezone differences when combined with rollover time variations can lead to a period of 2 days discrepancy. At New Year a blank table will be displayed during this period by default. For example, 30 or 31 (depending on whether rollover has occured) in December of the last year could be the latest available for a site in USA whilst the original version of the code is showing a blank table of the next year during early morning on 2 January in Austrailia. Even when station and viewer are in same timezone, there will be a blank table until the first rollover on 2 January. |
|||
*Version 1.2 defaults to showing the year found in the last row of dayfile.txt. |
|||
*The final field on each row is not read correctly, because the row break has been specified wrongly for a Microsoft Windows environment, it should quote '''fdata.split("\r\n");''' i.e. ''defines that carriage return then line feed (newline) used to split daily observations in dayfile''. Because the script presented here only processes a small number of fields from earlier in the row, this error does not affect the output for the published code. |
|||
* See more background in forum thread [http://sandaysoft.com/forum/viewtopic.php?f=18&t=8600 AnnualDataSummary PHP]. |
|||
=== Configuration Variables === |
|||
By default in 'readDayfile.php': |
|||
* Assume the dayfile.txt is read from a subfolder called 'data'. |
|||
** Look for the line 'dayfile='/data/dayfile.txt';' around line 15 and change this to point to your dayfile.txt on your webserver. |
|||
:for ''PHP'' version the path should start from the / (root) |
|||
== Another approach using a combination of PHP and jQuery == |
|||
* Assume the date format in the dayfile is ''dd/mm/yy'' |
|||
** You may need to edit the file if you do not use '/' between the parts of the date (the month is always in the middle for all Cumulus log files), look in file for ''date_delimiter''' - the symbol separating your date format. See [[setup]]. |
|||
''Unsure who supplied this approach, need to do a forum search'' |
|||
* Assume the data in the dayfile is separated with a ''comma'' |
|||
** Look for the line mentioning '''field_delimiter''' - the symbol separating each of your fields in the dayfile.txt. For most people this is a '''comma''' but (if you use comma to separate integer and decimal parts of real numbers) it could be a semi-colon (''';''') or other symbol. |
|||
# extract all the values off the daily summary table with a simple "SELECT * FROM name_of_your_table" query |
|||
# store those values in XML format using a PHP script called 'read_dayfile_historic.php' ''(needs a reference to say where get this)'' |
|||
# use a jQuery library routine to produce a table of values using JavaScript: |
|||
<pre> |
|||
<script src="jquery.jqGrid.min.js" type="text/javascript"></script> |
|||
<script type="text/javascript"> |
|||
jQuery(document).ready(function(){ |
|||
jQuery("#list").jqGrid({ |
|||
url:'read_dayfile_historic.php', |
|||
datatype: 'xml', |
|||
height: 460, |
|||
width: 970, |
|||
mtype: 'GET', |
|||
colNames:['Date','Highest<br/>Gust<br/>(mph)','Wind<br/>Bearing','Time of<br/>Gust','Min. Temp<br/>(°C)','Time of<br/>Min.','Max. Temp<br/>(°C)','Time of<br/>Max.','Avg. Temp<br/>(°C)','Lowest<br/>pressure<br/>(mb)','Time of<br/>low','Highest<br/>pressure<br/>(mb)','Time of<br/>high','Highest<br/>Rain rate<br/>(mm/hr)','Time of<br/>High Rate','Total Rain<br/>fall (mm)','Tot.Wind<br/>run'], |
|||
colModel :[ |
|||
{name:'UKLogDate', index:'LogDate', width:65, label: 'Date', sortable: true}, |
|||
{name:'HighestWindGust', index:'HighWindGust', width:50, align:'right'}, |
|||
{name:'HWindGBear', index:'HWindGBear', width:50, align:'right'}, |
|||
{name:'THWindG', index:'THWindG', width:55, align:'right', sortable: false}, |
|||
{name:'MinTemp', index:'MinTemp', width:55, align:'right'}, |
|||
{name:'TMinTemp', index:'TMinTemp', width:55, align:'right', sortable: false}, |
|||
{name:'MaxTemp', index:'MaxTemp', width:60, align:'right'}, |
|||
{name:'TMaxTemp', index:'TMaxTemp', width:55, align:'right', sortable: false}, |
|||
{name:'AvgTemp', index:'AvgTemp', width:57, align:'right'}, |
|||
{name:'MinPress', index:'MinPress', width:60, align:'right'}, |
|||
{name:'TMinPress', index:'TMinPress', width:50, align:'right', sortable: false}, |
|||
{name:'MaxPress', index:'MaxPress', width:60, align:'right'}, |
|||
{name:'TMaxPress', index:'TMaxPress', width:50, align:'right', sortable: false}, |
|||
{name:'MaxRainRate', index:'MaxRainRate', width:55, align:'right'}, |
|||
{name:'TMaxRR', index:'TMaxRR', width:60, align:'right', sortable: false}, |
|||
{name:'TotRainFall', index:'TotRainFall', width:55, align:'right'}, |
|||
{name:'TotWindRun', index:'TotWindRun', width:55, align:'right'}], |
|||
pager:'#pager', |
|||
rowNum:19, |
|||
rowList:[10,20,30], |
|||
viewrecords: true, |
|||
sortname: "LogDate", |
|||
sortorder: "desc", |
|||
caption: 'Daily Statistics' |
|||
}); |
|||
}); |
|||
</script> |
|||
</pre> |
|||
The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them. The selection of column names used above are from the schema created by David Jamieson, as used by Cumulus MX. |
|||
==Developing your own script?== |
|||
If you have the technical computer script language knowledge to write your own script for reading "dayfile.txt", then you do need some understanding of how Steve Loft developed that file. The forum post [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=2367 a PHP script to read the contents of the dayfile.txt] is a good starting point, it is about the development of [[ImportCumulusFile]]. |
|||
That post explains the date delimiter, how a Comma Separated Variable file may use other punctuation to separate the fields, how to specify the location of your dayfile.txt log to the script and the reason the key variable is used. |
|||
The thread contains several insights into how Cumulus was developed and it is useful to read Steve's posts in that thread if you are developing your own scripts. |
|||
If, having developed a script, you want to share it with others, publish it on the Cumulus Support Forum in the '''Web general''' sub forum. However, be prepared to deal with ongoing support, writing a script that will work out of the box for other people is impossible, there are so many ways that people set up their website. For example if your script accesses a database, be aware that not everyone has the same approach to missing data. Decide whether your script will only work with perfect data, whether you need to treat zeroes as missing data, or whether your script expects missing data to have "Null" in the relevant column. As MX is developed and supports a greater variety of sensors, you cannot assume that everybody uses the column names invented by David Jamieson and used by Cumulus MX, some people may want their database table to use daily derivatives that do not come from a field in dayfile.txt, a few might want to implement a more consistent way of naming columns. If your script uses jQuery, or PHP, be aware that the versions you have done your development on may not match the versions used by other people. Generally anything written in just HTML and JavaScript (despite evolving ECMA 262 standard) will be portable, everyone now uses HTML 5, CSS 3, and even Microsoft browsers use JavaScript, not their own jSCript (used by early Internet Explorer versions). |
|||
At the end of this Wiki page are several sections dealing with issues found with the earliest script that was developed, even the original author ditched that script, and adopted one written by another author. That said the example sites section shows that at least one site is still using that script, at a later version. |
|||
==Key differences between the legacy Cumulus 1 and MX== |
|||
Cumulus 1 only reads "dayfile.txt" when you choose to display a period, or edit the file. MX reads the entire file when it starts and uses the file for various charting operations including [[Highcharts - Historic|history chart drawing]]. Thus with MX it is more important to take a copy of the file once a day and use that copy for all the scripts discussed on this Wiki page. |
|||
MX has (optional) functionality to create/update a database table, that by default uses columns that are all derived from fields in dayfile.txt; if you choose the defaults for this table creation then another option will update the table if you use the editor on dayfile.txt. (You can choose to give the table a different name or different columns, MX provides Custom options to automatically upload (insert rows,replace rows, update particular columns) at various time intervals for a table with different columns, but it won't do this when you edit the file if either table name or table columns have changed). |
|||
Cumulus 1 does not include any functionality for creating/updating a database table to hold daily summarised data. You will find below various suggestions about how to get a copy of your file to your web server, and how to update a daily based table on your database server |
|||
=Example sites= |
|||
Do be aware that references to external sites either in the Wiki or the Forum may quote links that no longer work, the destination site might have closed down, changed its name, or just eliminated its past content in a revamp. |
|||
If you do want access to a script, or demonstration, that you can no longer find, post in the forum as it might still exist in an archive somewhere, or another Cumulus user may have a copy to share with you. |
|||
== Third-party contributions == |
|||
Here are some URLs where some of the scripts in the list earlier can be viewed. |
|||
# This Wiki page was inspired by the David Jamieson (DAJ) code. Its problems are discussed later on this Wiki page, and nobody uses it now. |
|||
# An improved version of that was developed by BCJKiwi, and a working example of that Annual Weather Data Summary Ver 1.2. November 2012 Script (PHP script) can still be found [http://www.fordingbridgeweather.co.uk/datasummary.php here] |
|||
#* A different carrier page (see below) is the Weather Blue set of Alternative Templates available from the French only site [http://meteoduquebec.com/ Météo du Québec] and the international site [http://weatherbyyou.com/demo.php <strong>Weather</strong> <small>by</small><sub>You</sub>]. The link that used to be here for a daily summary page has been removed as that site no longer uses it. |
|||
# David's site https://www.fordingbridgeweather.co.uk/datasummary.php, now uses an improved version of the 'SQL dayfile data Viewer' script . |
|||
# There are many websites using a derivative of the YADR developed by beteljuice, including https://weather.wilmslowastro.com/datasummary.php and by selecting Reports, then YADR at https://meteo-wagenborgen.nl |
|||
''Please update or add to these links if you find other exampless.'' |
|||
=Getting a copy of dayfile.txt onto web server= |
|||
If you are using Cumulus MX, the "Extra Web Files" settings page allows you to copy or file transfer at end of day. Set the local file to "data/dayfile.txt" (or if using Microsoft operating system "data\dayfile.txt"), remote file to destination, tick EOD column. |
|||
If you are using legacy Cumulus 1, use [[Upload_Dayfile|the method described here]] or [[Toolbox|Cumulus Toolbox]] to get a copy of the local file to use on their web server. (If your web server is local, you can simply copy the file to that server, once a day). |
|||
= Populating a database table = |
|||
*[[File:Badge v1.png]] The [[ImportCumulusFile]] page describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have been edited for MX, so you will need to use an older version of that script that matches the version of Cumulus 1 you are using. |
|||
*[[File:Badge vMx.png]] |
|||
** '''CumulusMX.exe''' has an optional feature with a standard option to insert a new row into a database table at end of each day - see [[MX_Administrative_Interface#Standard_Daily_Summary_Table]] section for details of how database table holding columns relating to the dayfile.txt fields. |
|||
** '''ExportToMySQL.exe''' (see [[Software#ExportToMySQL]]), can update the database table with past rows. |
|||
=Technical information= |
|||
== Displaying daily statistics from dayfile.txt == |
|||
If you choose a script that parses a copy of the [[Dayfile.txt]], then the script you choose might require some customisation: |
|||
* to define where the file is to be found, |
|||
* to define what character separates fields in the file, it does not have to be [[:Category:Files with Comma Separated Values|Comma Separated Values]] |
|||
* to define what character separates the day of month from month and from year |
|||
Some scripts might work out at least some of these from examining the file. |
|||
== Displaying daily statistics from a database table == |
|||
Cumulus MX has a standard name and standard set of columns it uses for the database table with one row for each day. If you have accepted these defaults, skip the next section, as it is likely that you are using a routine that knows the defaults and does not need any customisation for running against your database server. |
|||
If you use Cumulus 1, or you have created a database table with a different table name or with columns that suit you, then it is likely you have some database knowledge and skills, but the next section goes into some technical detail that might help you, although novices might want to skip it! |
|||
=== Schema === |
|||
# The retrieval script needs to have the following defined for it to use (it is suggested that these variables are defined in a separate PHP script that is in a directory that the PHP parser can access, but is not in the part of the web server that is visible to a client): |
|||
#* The server host name, or w.x.y.z format Internet Protocol (=IP) address, running your MySQL database |
|||
#* The user name used to log-in to your database server |
|||
#* The password used to log-in to your database server |
|||
#** (It is good practice to have separate authentication for users with privileges of read only access, and for users with privileges of read and update access, alternatively use an additional key variable as David does in his update script described below) |
|||
#* The name of the (MySQL or whatever) database with the table |
|||
#* The name for the daily summary table |
|||
# The retrieval script needs to know the names of the columns it is to read from the database. |
|||
#* Database designers talk about the ''schema'' meaning the specification that describes the design of the table. |
|||
#* A schema lists the column names together with what type and size of variables they hold. |
|||
#* For the daily summary database table, the columns can copy the fields in the [[dayfile.txt|daily summary log file]], and this is what David Jamieson does in his approach [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=2367 a PHP script to read the contents of the dayfile.txt] where the columns are generated in the same order as the corresponding fields in the log file, so David's script uses the field numbers to identify the columns by Position (rather than by Column Name). |
|||
#* Remember that while the fields in the log appear in the order that statistic was added during Cumulus 1 development, in theory the columns can appear in any order (if all your scripts identify the columns by name rather than position) so columns associated with rain (these are separated in dayfile.txt fields 12, 13, 14, 31, and 32) can appear together, as can those for wind (these are dispersed in dayfile.txt fields 1, 2, 3, 16, 17, 18, and 39). |
|||
# If you have the skills to write your own script, then you might want to consider a different schema, daily information is in three log files for Cumulus 1, "dayfile.txt" should have the statistics for all completed days (that excludes today), "today.ini" has values for today-so-far, and the [[Weather Diary]] where snow records (and general weather text) can be manually recorded. A schema that combines information extracted from those three files to show everything the legacy Cumulus has for a day is available in [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=16880#p129719 this forum post]. |
|||
==== Alternative schemas ==== |
|||
Of course, the schema in your database table does not need to exactly mimic the fields in the dayfile.txt file. |
|||
* Perhaps you do not use all of the fields; your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table |
|||
* Perhaps you want columns representing values from extra sensors or other log files. |
|||
* Functionality is included in Cumulus MX to handle your own schema (your choice of columns); for full details of the functionality see [https://cumulus.hosiene.co.uk/viewtopic.php?f=27&t=12908&start=15#p107639 Cumulus MX Announcements - PLEASE READ THESE!]. |
|||
===Validation=== |
|||
The standard functionality in MX, and that described in [[ImportCumulusFile]], assume that your data within Cumulus is perfect, you never have rogue values, nor do you have any missing data, and that you are using a weather station that has all the sensors to populate all the fields in dayfile.txt. |
|||
Thus your database table gets an exact copy of what numbers were found for all values and time-stamps in the logging file. Each of your scripts driving web pages for viewing data from your database table needs to include some validation to ensure that what is shown makes sense. |
|||
Alternatively, you might feel that you want to validate all the fields read from the daily summary file, before writing them to your database table to ensure your database only contains good quality data. Equally, to simplify querying your database table, you might want missing data to be stored as real nulls in the database columns, rather than the "zero" or "space" or empty character string or even sometimes 99.99 or -99, that Cumulus MX uses in its inconsistent way. That way you don't get any longer period averages skewed by the invalid numbers. |
|||
To implement those changes, you need a little knowledge about Structured Query Language, and you will need to write your own script (possibly based on the script in [[ImportCumulusFile]], that validates what it reads from each field, and stores 'NULL' for any unacceptable content. Having ensured your database table only has good data, all your retrieval queries can much more easily ensure that you don't get what is shown on your web pages skewed by invalid data. |
|||
=The Inspiring Script= |
|||
The rest of this Wiki page can be skipped, unless you have a thirst for knowledge about the development of early scripts for processing this daily information. The text is retained purely as historic record, but if you are developing your own script you might want to understand what problems were hit earlier. For PHP scripts a lot of the issues faced by authors of scripts derive from the way that new releases of PHP add new features and deprecate others, this leads to better functionality but it can stop old scripts working! |
|||
== Issues with DAJ script == |
|||
The technical issues and the usability issues below, have been addressed by BCJKiwi's replacement script, by the SFWS alternative script, and by the beteljuice "Yet Another Dayfile Reader". |
|||
The following sections describe how DAJ's script made configuration complex. |
|||
=== Localisation / Language in DAJ script === |
|||
DAJ claimed that his script had been designed to be easily translated to your language of choice. |
|||
=== Localization / Language === |
|||
The script has been designed to be easily translated to your language of choice. |
|||
Just edit the readDayfile.php |
|||
*variable '''mn''' is a list of the 12 months of the year, in an abbreviated format. You may change these as necessary, but try to keep it to an abbreviation as there is limited space. |
*variable '''mn''' is a list of the 12 months of the year, in an abbreviated format. You may change these as necessary, but try to keep it to an abbreviation as there is limited space. |
||
Line 61: | Line 233: | ||
*The first element, in this case 'maxtemp' -- is a system variable to identify the data set, used for the URL parameter option, used by the ''Switch'' coding (so any change here needs to be reflected there), and also tested elsewhere in the code (in the Rainfall and Windgust data sets a particular style is applied to their zero values). |
*The first element, in this case 'maxtemp' -- is a system variable to identify the data set, used for the URL parameter option, used by the ''Switch'' coding (so any change here needs to be reflected there), and also tested elsewhere in the code (in the Rainfall and Windgust data sets a particular style is applied to their zero values). |
||
=== Adding/removing data sets in DAJ script === |
|||
However, if you add a new data set, then your new first element should be a unique identifier. |
|||
If you add a new data set, then your new first element should be a unique identifier. |
|||
*Second, '''Max Temp''' is the text to be displayed in the ''button'' at the top of the table. You may change this to your own language |
*Second, '''Max Temp''' is the text to be displayed in the ''button'' at the top of the table. You may change this to your own language |
||
Line 67: | Line 241: | ||
*Third, '''Maximum Temperature''' is the text shown at the top of the table to ''describe the current data set''; again you may change this |
*Third, '''Maximum Temperature''' is the text shown at the top of the table to ''describe the current data set''; again you may change this |
||
*Finally, 'true' will display this button at the top; 'false' will hide it. Therefore, if you do not wish to allow users to jump to the 'Rainfall data set' change the 'true' to 'false' in the 'rainfall' element of the variable. (true/false MUST be lowercase) |
*Finally, 'true' will display this button at the top; 'false' will hide it. Therefore, if you do not wish to allow users to jump to the 'Rainfall data set' change the 'true' to 'false' in the 'rainfall' element of the variable. (true/false MUST be lowercase) |
||
<span style="color: red">NOTE: [[User:Sfws|Sfws]] 12:22, 31 December 2012 (UTC)</span> |
|||
<span style="color: red">Having made any change to the array as DAJ says above, you also need to change what is in a '''switch''' section and you will find that is below where DAJ says '// Nothing to edit below here'.</span> |
<span style="color: red">Having made any change to the array as DAJ says above, you also need to change what is in a '''switch''' section and you will find that is below where DAJ says '// Nothing to edit below here'.</span> |
||
Line 79: | Line 251: | ||
=== Instructions === |
|||
The instructions in [[AnnualDataSummary| David's original]] article for PHP version still apply, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant. |
|||
== Update of DAJ PHP script by BCJKiwi == |
|||
*Download the following zip file...[http://wiki.sandaysoft.com/files/AnnualDataSummary_12.zip AnnualDataSummary_12.zip], the zip contains 3 files: |
|||
There is a revised set of files (PHP only) ver 1.2. contributed by BCJKiwi on 17 December 2012 that update David's Ver 1.2 (November 2012) fileset to eliminate errors in validation due to non-standard or deprecated code routines. They also include a couple of updates from DAJ version as advised by posts in the Forum. The text that follows describes issues with David's script, and how they were rectified by Brian's script |
|||
See [[AnnualDataSummary]] page. |
|||
=== Configuration Variables === |
|||
One issue with early scripts was that the user had to edit the files in the package to configure the script to work in their installation, here are details of the sort of issues end users had with the assumptions in those scripts. |
|||
By default in 'readDayfile.php': |
|||
* Assume the dayfile.txt is read from a subfolder called 'data'. |
|||
** Look for the line 'dayfile='/data/dayfile.txt';' around line 15 and change this to point to your dayfile.txt on your webserver. |
|||
:for ''PHP'' version the path should start from the / (root) |
|||
* Assume the date format in the dayfile is ''dd/mm/yy'' |
|||
** You may need to edit the file if you do not use '/' between the parts of the date (the month is always in the middle for all Cumulus log files), look in file for ''date_delimiter''' - the symbol separating your date format. See [[setup]]. |
|||
* Assume the data in the dayfile is separated with a ''comma'' |
|||
** Look for the line mentioning '''field_delimiter''' - the symbol separating each of your fields in the dayfile.txt. For most people this is a '''comma''' but (if you use comma to separate integer and decimal parts of real numbers) it could be a semi-colon (''';''') or other symbol. |
|||
=== Errors rectified by Brian's version === |
|||
*Non-standard attributes 'datacol' and 'datarow' are generated by the original javascript and php routines that will not pass validation against standards (in HTML5 these can be made to validate by changing to 'data-col' and 'data-row'). |
|||
*In version 1.1 of David's script, jQuery version 1.5 is called, this permitted use of '.attr' object to add/remove the highlight; |
|||
** this will not work for later jQuery version 1.6 asuse of that object was deprecated and the '.prop' object is now specified for the same purpose. |
|||
*Version 1.0 and 1.1 of David's script default to showing a table for viewer's local current calendar year. Remember the dayfile.txt (even if updated each day) only contains records up to the day ending at the last rollover time using the timezone local to the weather station. Timezone differences when combined with rollover time variations can lead to a period of 2 days discrepancy. At New Year a blank table will be displayed during this period by default. For example, 30 or 31 (depending on whether rollover has occurred) in December of the last year could be the latest available for a site in USA whilst the original version of the code is showing a blank table of the next year during early morning on 2 January in Australia. Even when station and viewer are in same timezone, there will be a blank table until the first rollover on 2 January. |
|||
*Version 1.2 of David's script defaults to showing the year found in the last row of dayfile.txt. |
|||
*The final field on each row is not read correctly, because the row break has been specified wrongly for a Microsoft Windows environment, it should quote '''fdata.split("\r\n");''' i.e. ''defines that carriage return then line feed (newline) used to split daily observations in dayfile''. Because the script presented here only processes a small number of fields from earlier in the row, this error does not affect the output for the published code. |
|||
* See more background in forum thread [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=8600 AnnualDataSummary PHP]. |
|||
=== Instructions for using any of these scripts === |
|||
The instructions on [[AnnualDataSummary| David's original]] Wiki page for PHP version still apply to most subsequent alternatives, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant. |
|||
*Download the following zip file...[https://cumuluswiki.org/files/AnnualDataSummary_12.zip AnnualDataSummary_12.zip], the zip contains 3 files: |
|||
*#'''readDayfile.php''' - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style |
*#'''readDayfile.php''' - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style |
||
*#'''datasummary.php''' - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) ''Settings.php'' and ''common.php''. You can use your own page instead of this file - see [[#Creating_your_own_Carrier_Page|carrier page customisation]] below. |
*#'''datasummary.php''' - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) ''Settings.php'' and ''common.php''. You can use your own page instead of this file - see [[#Creating_your_own_Carrier_Page|carrier page customisation]] below. |
||
Line 92: | Line 300: | ||
=== Usage === |
=== Usage === |
||
* Open ''datasummary.php'' from your website in a browser |
* Open ''datasummary.php'' from your website in a browser |
||
Line 99: | Line 308: | ||
=== Opening a specific data set when the page is first loaded === |
=== Opening a specific data set when the page is first loaded === |
||
By default (in the original version) the table created will show the available daily values of the ''maximum temperature'' for the current ''local calendar year'', however you can start with any data set and any year (assuming you have those values in the dayfile.txt) by ''adding a 'Query String' to the end of your URL'' in the browser. |
By default (in the original version) the table created will show the available daily values of the ''maximum temperature'' for the current ''local calendar year'', however you can start with any data set and any year (assuming you have those values in the dayfile.txt) by ''adding a 'Query String' to the end of your URL'' in the browser. |
||
Line 126: | Line 336: | ||
=== Creating your own Carrier Page === |
=== Creating your own Carrier Page === |
||
To create your own page, follow another page you are using successfully on your site and modify it by removing any content from <body> section you don't want and including in the <head> section: |
To create your own page, follow another page you are using successfully on your site and modify it by removing any content from <body> section you don't want and including in the <head> section: |
||
* the CSS styling: |
* the CSS styling: |
||
Line 139: | Line 350: | ||
=== Styling === |
=== Styling === |
||
The table styling is completely configurable using the included [[Stylesheet|Cascading Style Sheet]] datasummary.css. By default it is using similar colours to the standard Cumulus website. Below are a few of the key entries to consider when adjusting settings (typically colouring)... Also see [http://sandaysoft.com/forum/viewtopic.php?f=14&t=5319 support forum thread about colouring with php] |
|||
The table styling is completely configurable using the included [[Stylesheet|Cascading Style Sheet]] datasummary.css. By default it is using similar colours to the standard Cumulus website. Below are a few of the key entries to consider when adjusting settings (typically colouring)... Also see [https://cumulus.hosiene.co.uk/viewtopic.php?f=14&t=5319 support forum thread about colouring with php] |
|||
In the 'datasummary.css' file: |
In the 'datasummary.css' file: |
||
Line 172: | Line 384: | ||
[[Category:User Contributions]] |
|||
== Enhancement of JavaScript script == |
|||
This section was originated by [[User:Sfws|Sfws]] to mention script originated 24 Nov 2012 |
|||
Update of text here 00:53, 18 May 2013 (PDT) |
|||
Finalised version of script in forum 7:08 am 15 Jan 2014, minor update 8:51 pm 15 Nov 2014 |
|||
Find my package in [http://sandaysoft.com/forum/viewtopic.php?f=18&t=8690 alternative Javascript/jQuery solution] within the '''3rd party tools''' sub forum of the Cumulus forum. |
|||
I originally created the alternative Javascript version in December 2011 by editing the DAJ original (with similar aims to BCJKiwi with his PHP version 1.2) to offer easier customisation plus standards compliance, i.e. addressing all the issues [[AnnualDataSummary#Possible_problems|with David's version]]. If you don't use PHP, this is the script you should install, and you will find various threads on the forum about it in the Web site - General sub-forum as well as posts in the thread containing the source. |
|||
<span style="color: red">It really is much easier to add new data sets if you use this alternative JavaScript version in the forum. That includes more explanation about the original coding, and an alternative simpler way to specify and adjust the columns in a revised version of the array specified for the PHP version above.</span> |
|||
The advantages of my script are fully described in a file within the package. They include standards compliance, better integration with standard Cumulus pages, statistical additions to the bottom of each month's column, and easier tailoring to your preferences on wording, language, and text colouring. During 2012-2013, my alternative JavaScript version was rewritten from scratch (losing the original DAJ code), a number of additional features were added, whilst attempting to preserve ease of customisation to suit any requirements you have e.g. on the language and level of diagnostics output. One new feature is ability to scroll sideways, so calendar look can start with any month (any year) but always with 12 columns showing month headings. |
|||
One site using in January 2013 a variant based on an early version of my Javascript/jQuery alternative code, with its own look to the page, is [http://sassafras.id.au/projects/weather/dailyrecords.shtml found here]. Whilst a number of other people have tried later versions of this alternative code, and you can see some commenting in the forum thread, remember this was designed for easy tailoring, so is not an off the shelf package. |
|||
At the end of 2014, I rewrote this again, by adding use of PHP script for some of the actions previously done by JavaScript, retaining JavaScript for handling the selection of a change of criterion, and for handling the change of months to be shown (either by scrolling one month at a time or by picking one of the predefined periods on buttons). The resulting script was to prove very useful to me, but with the mix of languages (and the way it was developed) I found it was not easy to understand and hard to improve so I abandoned any idea of a version to share. However, I decided to rewrite that PHP version making it simpler by totally eliminating JavaScript, although given how many PHP versions existed (see below), mine was not shared. |
|||
== Variants of the original Daily Summary discussed in the support forum == |
|||
Many of the forum discussions are in the sub-forum: '''Cumulus > Cumulus 1 > 3rd Party Tools''', but there are a few in the sub-forum: |
|||
''Website Development > Web site - General''. So you really need to search through both for all available information as it is spread through hundreds of topics. I highlight just the most useful threads below. |
|||
=== Using JavaScript === |
|||
As well as the alternative JavaScript version above, SFWS produced a few other scripts that were developed from David's original script including [http://sandaysoft.com/forum/viewtopic.php?f=14&t=8631&hilit=one+year+ago#p71709 Comparing with one year ago], a script that allowed you to compare statistics from yesterday (in Cumulus' yesterdayT.htm style) with those from a year earlier (or any other past date). The forum thread includes feedback and consequent changes. |
|||
=== PHP scripts === |
|||
Look in the Website Development section of the Cumulus support forum - '''Web general''' sub forum: |
|||
* [http://saratoga-weather.org/legacy-scripts/dayfile.txt Source: Parse dayfile.txt PHP no database] - This script by the late Kevin Reed who ran the web site TNETweather.com, it is a no frills reader written in PHP (change '.txt' into '.php' when you download it) that parses dayfile.txt and produces an output table with headings. The script can be downloaded on Ken True's site from the link quoted, but see the forum reference [http://sandaysoft.com/forum/viewtopic.php?f=14&t=2419#p20155 Forum: Parse dayfile.txt PHP no database] for the only further information still available. |
|||
*[http://sandaysoft.com/forum/viewtopic.php?f=14&t=5319 'Annual Data Summary - Coloured Values'] - An extensively coloured version of David's original script with some useful discussion on the trade-off between testing values and efficient code here. |
|||
*[http://sandaysoft.com/forum/viewtopic.php?f=14&t=11133 'Annual data Summary (DAJ script) - show day-of-week'] - This is just one of several places on the forum where further ideas are explored by Mark Crossley and beteljuice. |
|||
*[http://sandaysoft.com/forum/viewtopic.php?f=14&t=11397&start=405#p91488 'Yet Another Dayfile Reader (PHP)'] - Download from here the ultimate PHP parsing dayfile.txt version that has progressed a long way from David's original, it incorporates all the ideas discussed in earlier forum topics, some of the features are: the highlighting makes day of week appear in row headers, you can swap between daily and seasonal views and there is a table summary including averages. See that forum url for links to some of the many sites that have adopted this version, if you do not have a database, but do have PHP, then this is version for you. Mark Crossley has his own offshoot of this script, [https://weather.wilmslowastro.com/datasummary.php?view=getorfmiland data summary], that reads a database table, but uses much of the same script so produces a similar summary. |
|||
* [http://sandaysoft.com/forum/viewtopic.php?f=14&t=16891#p130117 'Re: New Script that displays a selected daily statistic for every day'] - In [[#Enhancement of JavaScript script]] above, [[User:Sfws|Sfws]] refers to a pure PHP version that reads data from a database table. That version is now available in the linked paragraph, but using a site specific schema. |
|||
* [http://sandaysoft.com/forum/viewtopic.php?f=14&t=16891#p129796 'New Script that displays a selected daily statistic for every day'] - Download from here, in the above topic, the newest (at time of writing) way to display daily statistics. This is the "daily pick" script where you select one criterion and all the daily values available are output, for ease of display there is a separate table for each month with HTML <detail> tags to show/hide. Both this and previous scripts were written for a schema that is different to the David Jamieson/Mark Crossley/Cumulus MX standard, but the daily pick script has both schemas available, comment out the one you don't want and remove comments round the one you do want. See further information in posts in the topic and the topic that it follows on from, at [http://sandaysoft.com/forum/viewtopic.php?f=14&t=16880 MySQL Code Help]. |
|||
*[http://sandaysoft.com/forum/viewtopic.php?f=14&t=15983 'SQL dayfile data Viewer'] - This is another (widely adopted) PHP script for reading daily values from a database. It displays a number of key statistics, and you can order the results by the contents of any column. Read the forum topic for more information. You can scroll through the days or set a filter to only show certain days. There is a summary line, but because some columns include units, the calculation of the summary for those columns fails. It is quite simple to edit out the units, so all columns show a summary, but most users adopt the script as published and just accept that some summary figures are replaced by text. |
|||
== Displaying daily statistics from a database table == |
|||
=== Schema === |
|||
# The retrieval script needs to have the following defined for it to use (it is suggested that these variables are defined in a separate PHP script that is in a directory that the PHP parser can access, but is not in the part of the web server that is visible to a client): |
|||
#* The server host name, or w.x.y.z format Internet Protocol (=IP) address, running your MySQL database |
|||
#* The user name used to log-in to your database server |
|||
#* The password used to log-in to your database server |
|||
#** (It is good practice to have separate authentication for users with privileges of read only access, and for users with privileges of read and update access, alternatively use an additional key variable as David does in his update script described below) |
|||
#* The name of the (MySQL or whatever) database with the table |
|||
#* The name for the daily summary table |
|||
# The retrieval script needs to know the names of the columns it is to read from the database. |
|||
#* Database designers talk about the ''schema'' meaning the specification that describes the design of the table. |
|||
#* A schema lists the column names together with what type and size of variables they hold. |
|||
#* For the daily summary database table, the columns can copy the fields in the [[dayfile.txt|daily summary log file]], and this is what David Jamieson does in his approach [http://sandaysoft.com/forum/viewtopic.php?f=18&t=2367 a PHP script to read the contents of the dayfile.txt] where the columns are generated in the same order as the corresponding fields in the log file, so David's script uses the field numbers to identify the columns by Position (rather than by Column Name). |
|||
#* Remember that while the fields in the log appear in the order that statistic was added during Cumulus 1 development, in theory the columns can appear in any order (if all your scripts identify the columns by name rather than position) so columns associated with rain (these are separated in dayfile.txt fields 12, 13, 14, 31, and 32) can appear together, as can those for wind (these are dispersed in dayfile.txt fields 1, 2, 3, 16, 17, 18, and 39). |
|||
# If you have the skills to write your own script, then you might want to consider a different schema, daily information is in three log files for Cumulus 1, "dayfile.txt" should have the statistics for all completed days (that excludes today), "today.ini" has values for today-so-far, and "log.xml" is the Weather Diary where snow records (and general weather text) can be manually recorded. A schema that combines information extracted from those three files to show everything Cumulus has for a day is available in [http://sandaysoft.com/forum/viewtopic.php?f=14&t=16880#p129719 this forum post]. |
|||
=== Loading and Updating the database table === |
|||
Obviously if you are using a script that reads statistics from a table, you also need to be running a script that ensures the table is updated at the end of each day, and therefore always up to date. |
|||
* Functionality is included in Cumulus MX to do this update, it uses the schema devised by Mark Crossley by default, but Cumulus MX can also handle your own schema; for full details of the functionality see [http://sandaysoft.com/forum/viewtopic.php?f=27&t=12908&start=15#p107639 Cumulus MX Announcements - PLEASE READ THESE!]. |
|||
* Cumulus 1 users need to use a script to create a database table, populate and update it from the daily summary log file. The most widely used script is an update by Mark Crossley of the script mentioned above by David Jamieson; and it is described in [[ImportCumulusFile|this Wiki article]]. |
|||
** It is however worth also reading the forum post [http://sandaysoft.com/forum/viewtopic.php?f=18&t=2367 a PHP script to read the contents of the dayfile.txt] as that explains the date delimiter, how a Comma Separated Variable file may use other punctuation to separate the fields, how to specify the location of your dayfile.txt log to the script and the reason the key variable is used. The thread contains several insights into how Cumulus was developed and it is a definite must to read Steve's posts in that thread if you are developing your own scripts. |
|||
=== Some example Scripts === |
|||
Look in the Website Development section of the Cumulus support forum - '''Web general''' sub forum: |
|||
* Look at the Javascript and PHP scripts already listed for showing raw values, but some others present a summary view: |
|||
**[http://sandaysoft.com/forum/viewtopic.php?f=14&t=10755#p87081 Top 10 Records Page]- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. ). The latest source php can be seen using a [http://weather.wilmslowastro.com/top10s.php?view=sce query-string on Mark Crossley's web site] and there have been several updates since the forum thread was created. |
|||
**[http://sandaysoft.com/forum/viewtopic.php?f=14&t=16534#p126964 What to do with data from MySQL] - This posting lists a number of web pages on PaulMy's web site where he presents statistics from the daily summary table of his database in different formats. |
|||
**[http://sandaysoft.com/forum/viewtopic.php?f=14&t=7834#p66013 This day last year page] - This post has a query in a php file (as you save the download change '.txt' to '.php') that will extract from the daily summary table in a database, values for one year ago, two years ago to feed a web page that displays all the summary statistics for today and the corresponding ones for earlier years. |
|||
There are various scripts out there that output graphs from the values read, I have not identified a particular one to reference here yet. If you find a site dislaying a graph you would like to copy on your own site, you might be able to see the PHP source that is responsible, but as I have commented [http://sandaysoft.com/forum/viewtopic.php?f=14&t=16425&p=126065#p126065 elsewhere] most authors do not make it easy to see their code. Of course one of the reasons for using PHP script is that processing is done quickly on the web server and only the resulting HTML is passed onto the client browser and this does protect interlectual property and adds security by not showing the database access code. |
|||
=== Other jQuery approaches === |
|||
If you extract all the values off the daily summary table with a simple "SELECT * FROM name_of_your_table" query and store those values in XML format using a PHP script called 'read_dayfile_historic.php', then you can use a jQuery library routine to produce a table of values using JavaScript: |
|||
<pre> |
|||
<script src="jquery.jqGrid.min.js" type="text/javascript"></script> |
|||
<script type="text/javascript"> |
|||
jQuery(document).ready(function(){ |
|||
jQuery("#list").jqGrid({ |
|||
url:'read_dayfile_historic.php', |
|||
datatype: 'xml', |
|||
height: 460, |
|||
width: 970, |
|||
mtype: 'GET', |
|||
colNames:['Date','Highest<br/>Gust<br/>(mph)','Wind<br/>Bearing','Time of<br/>Gust','Min. Temp<br/>(°C)','Time of<br/>Min.','Max. Temp<br/>(°C)','Time of<br/>Max.','Avg. Temp<br/>(°C)','Lowest<br/>pressure<br/>(mb)','Time of<br/>low','Highest<br/>pressure<br/>(mb)','Time of<br/>high','Highest<br/>Rain rate<br/>(mm/hr)','Time of<br/>High Rate','Total Rain<br/>fall (mm)','Tot.Wind<br/>run'], |
|||
colModel :[ |
|||
{name:'UKLogDate', index:'LogDate', width:65, label: 'Date', sortable: true}, |
|||
{name:'HighestWindGust', index:'HighWindGust', width:50, align:'right'}, |
|||
{name:'HWindGBear', index:'HWindGBear', width:50, align:'right'}, |
|||
{name:'THWindG', index:'THWindG', width:55, align:'right', sortable: false}, |
|||
{name:'MinTemp', index:'MinTemp', width:55, align:'right'}, |
|||
{name:'TMinTemp', index:'TMinTemp', width:55, align:'right', sortable: false}, |
|||
{name:'MaxTemp', index:'MaxTemp', width:60, align:'right'}, |
|||
{name:'TMaxTemp', index:'TMaxTemp', width:55, align:'right', sortable: false}, |
|||
{name:'AvgTemp', index:'AvgTemp', width:57, align:'right'}, |
|||
{name:'MinPress', index:'MinPress', width:60, align:'right'}, |
|||
{name:'TMinPress', index:'TMinPress', width:50, align:'right', sortable: false}, |
|||
{name:'MaxPress', index:'MaxPress', width:60, align:'right'}, |
|||
{name:'TMaxPress', index:'TMaxPress', width:50, align:'right', sortable: false}, |
|||
{name:'MaxRainRate', index:'MaxRainRate', width:55, align:'right'}, |
|||
{name:'TMaxRR', index:'TMaxRR', width:60, align:'right', sortable: false}, |
|||
{name:'TotRainFall', index:'TotRainFall', width:55, align:'right'}, |
|||
{name:'TotWindRun', index:'TotWindRun', width:55, align:'right'}], |
|||
pager:'#pager', |
|||
rowNum:19, |
|||
rowList:[10,20,30], |
|||
viewrecords: true, |
|||
sortname: "LogDate", |
|||
sortorder: "desc", |
|||
caption: 'Daily Statistics' |
|||
}); |
|||
}); |
|||
</script> |
|||
</pre> |
|||
The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them. The column names used above are from the schema by David Jamieson, as used by Cumulus MX. |
Latest revision as of 12:29, 9 January 2022
DAILY SUMMARY
Available options
This page describes routines that either read Dayfile.txt log file, or run queries against a database table with one row per day. The various scripts then display either a selection of the rows, or a selection of the derivatives in a more friendly way that just browsing the file or table. In general, these scripts display some summary information, such as highest/lowest in month/season/year, and some scripts use coloured fonts to group values.
There is discussion about many of the available scripts in the Cumulus Support forum section called "Website Development", mostly (but not exclusively) in 3rd Party Tools sub-forum, and/or Web site - general subforum. Where there is information in this Wiki, links are given in summary below.
The scripts use various computer languages:
- C# - see Yadr, (a novice might prefer this first choice as it is part of a package, so there is no code to edit)
- jQuery - a number of scripts are available, if you make use of jQuery including:
- see the Annual Data Summary page contributed by David Jamieson (current web site) in March 2011, (be aware that this requires some user configuration (see towards end of this Wiki page), so some understanding of JavaScript helps, and you need to load jQuery).
- see Alternative Javascript/jQuery solution within the 3rd party tools sub forum of the Cumulus forum. (The advantages are discussed in file within December 2011 package, but essentially this has similar output to previous, but is far easier to customise)
- see Comparing with one year ago written in style of the yesterday web page provided with legacy Cumulus 1.
- PHP - a number of scripts are available (if you use Your Own Server then you need to install PHP), including:
- Be aware, some of these scripts only work with the syntax available in the PHP version that was current when the script was written
- Be aware, scripts may use "include ..." or "require ..." to bring in variables (such as those for accessing the database) or functions (to do standard actions) or templates (shared headers, navigation menus and footers) or text (any other common content) that is used with a number of pages on author's site. If you don't comment out those references to external scripts, or replace the references with your equivalents, the PHP parser may fail to produce a HTML page and nothing will be available to the browser. Thus you do need a simple understanding of PHP, a PHP book, or the ability to look up PHP instructions on the web if you are to be successful with tailoring these scripts to your site.
- If you find a site displaying a graph or table you would like to copy on your own site, you might be able to see the PHP source that is responsible, but (see here) most authors do not make it easy to see their PHP code. Of course one of the reasons for using PHP script is that processing is done quickly on the web server and only the resulting HTML is passed onto the client browser. This inability to see the PHP code that generates the web page does protect intellectual property and adds security against hacking by not showing the database access code.
- 'Yet Another Dayfile Reader (PHP)' - Download from here the ultimate PHP parsing dayfile.txt version that has progressed a long way from David's original, it incorporates all the ideas discussed in earlier forum topics, some of the features are: the highlighting makes day of week appear in row headers, you can swap between daily and seasonal views and there is a table summary including averages. See that forum url for links to some of the many sites that have adopted this version, if you do not have a database, but do have PHP, then this is version for you. Mark Crossley has his own offshoot of this script, data summary, that reads a database table, but uses much of the same script so produces a similar summary.
- Source: Parse dayfile.txt PHP no database - This script by the late Kevin Reed who ran the web site TNETweather.com, it is a no frills reader written in PHP (change '.txt' into '.php' when you download it) that parses dayfile.txt and produces an output table with headings. The script can be downloaded on Ken True's site from the link quoted, but see the forum reference Forum: Parse dayfile.txt PHP no database for the only further information still available.
- 'Annual Data Summary - Coloured Values' - An extensively coloured version of David's original script modified by Mark Crossley with some useful discussion on the trade-off between testing values and efficient code here. The source link in that thread no longer works, but (at time of typing this) the source may be found here on Paul's site.
- 'Annual data Summary (DAJ script) - show day-of-week' - This is just one of several places on the forum where further ideas are explored by Mark Crossley and beteljuice.
- 'Re: New Script that displays a selected daily statistic for every day' - In #Enhancement of JavaScript script above, Sfws refers to a pure PHP version that reads data from a database table. That version is now available in the linked paragraph, but using a site specific schema.
- 'New Script that displays a selected daily statistic for every day' - Download from here, in the above topic, the newest (at time of writing) way to display daily statistics. This is the "daily pick" script where you select one criterion and all the daily values available are output, for ease of display there is a separate table for each month with HTML <detail> tags to show/hide. Both this and previous scripts were written for a schema that is different to the David Jamieson/Mark Crossley/Cumulus MX standard, but the daily pick script has both schemas available, comment out the one you don't want and remove comments round the one you do want. See further information in posts in the topic and the topic that it follows on from, at MySQL Code Help.
- 'SQL dayfile data Viewer' - This is another (widely adopted) PHP script for reading daily values from a database. It displays a number of key statistics, and you can order the results by the contents of any column. Read the forum topic for more information. You can scroll through the days or set a filter to only show certain days. There is a summary line, but because some columns include units, the calculation of the summary for those columns fails. It is quite simple to edit out the units, so all columns show a summary, but most users adopt the script as published and just accept that some summary figures are replaced by text.
- Top 10 Records Page- This script by Mark Crossley runs several queries against the daily summary table of a database and picks various extremes displaying the top ten extremes (e.g. lowest daily temperature).
- What to do with data from MySQL - This posting lists a number of web pages on PaulMy's website where he presents statistics from the daily summary table of his database in different formats.
- Comparing with one year ago - This post as requested here has a query in a php file (as you save the download change '.txt' to '.php') that will extract from the daily summary table in a database, values for one year ago, to display alongside for example statistics for yesterday. An equivalent script using jQuery to read dayfile.txt directly was listed earlier. If you have some programming skills, you can vary either of these scripts to for example display statistics for each day in the last week (or month). I have written scripts that display the statistics for last week using a combination of this daily table in database approach and Cumulus recent tags. I have another script that uses the daily table in the database to calculate the equivalent of what Cumulus 1 shows in View menu, this month selection, for any past month and adds the equivalent of what Cumulus this month web tags show and the resulting statistics are shown for various selectable mixtures of past months.
Graphical output from dayfile.txt
There are various scripts out there that output graphs from the values in recent history web tags, or values read off either a daily table in a database or dayfile.txt directly. There are two to highlight here:
- An alternative to the Cumulus 1 or Cumulus MX trend web page is described in this topic this can use all the sources mentioned.
- Highcharts (that cannot be used on a business website) are exploited by Mark Crossley in this topic; it is a very long topic, and discusses various issues to consider with using such data, as well as including the script. Cumulus MX now implements this both in its interface pages and its default web pages.
Another approach using a combination of PHP and jQuery
Unsure who supplied this approach, need to do a forum search
- extract all the values off the daily summary table with a simple "SELECT * FROM name_of_your_table" query
- store those values in XML format using a PHP script called 'read_dayfile_historic.php' (needs a reference to say where get this)
- use a jQuery library routine to produce a table of values using JavaScript:
<script src="jquery.jqGrid.min.js" type="text/javascript"></script> <script type="text/javascript"> jQuery(document).ready(function(){ jQuery("#list").jqGrid({ url:'read_dayfile_historic.php', datatype: 'xml', height: 460, width: 970, mtype: 'GET', colNames:['Date','Highest<br/>Gust<br/>(mph)','Wind<br/>Bearing','Time of<br/>Gust','Min. Temp<br/>(°C)','Time of<br/>Min.','Max. Temp<br/>(°C)','Time of<br/>Max.','Avg. Temp<br/>(°C)','Lowest<br/>pressure<br/>(mb)','Time of<br/>low','Highest<br/>pressure<br/>(mb)','Time of<br/>high','Highest<br/>Rain rate<br/>(mm/hr)','Time of<br/>High Rate','Total Rain<br/>fall (mm)','Tot.Wind<br/>run'], colModel :[ {name:'UKLogDate', index:'LogDate', width:65, label: 'Date', sortable: true}, {name:'HighestWindGust', index:'HighWindGust', width:50, align:'right'}, {name:'HWindGBear', index:'HWindGBear', width:50, align:'right'}, {name:'THWindG', index:'THWindG', width:55, align:'right', sortable: false}, {name:'MinTemp', index:'MinTemp', width:55, align:'right'}, {name:'TMinTemp', index:'TMinTemp', width:55, align:'right', sortable: false}, {name:'MaxTemp', index:'MaxTemp', width:60, align:'right'}, {name:'TMaxTemp', index:'TMaxTemp', width:55, align:'right', sortable: false}, {name:'AvgTemp', index:'AvgTemp', width:57, align:'right'}, {name:'MinPress', index:'MinPress', width:60, align:'right'}, {name:'TMinPress', index:'TMinPress', width:50, align:'right', sortable: false}, {name:'MaxPress', index:'MaxPress', width:60, align:'right'}, {name:'TMaxPress', index:'TMaxPress', width:50, align:'right', sortable: false}, {name:'MaxRainRate', index:'MaxRainRate', width:55, align:'right'}, {name:'TMaxRR', index:'TMaxRR', width:60, align:'right', sortable: false}, {name:'TotRainFall', index:'TotRainFall', width:55, align:'right'}, {name:'TotWindRun', index:'TotWindRun', width:55, align:'right'}], pager:'#pager', rowNum:19, rowList:[10,20,30], viewrecords: true, sortname: "LogDate", sortorder: "desc", caption: 'Daily Statistics' }); }); </script>
The above code uses jQuery to output a table showing all the statistics from the database, note that any column containing values can be clicked upon and the table will be resorted by those values, but any column containing time-stamps has the sorting set to false, so you cannot pick them. The selection of column names used above are from the schema created by David Jamieson, as used by Cumulus MX.
Developing your own script?
If you have the technical computer script language knowledge to write your own script for reading "dayfile.txt", then you do need some understanding of how Steve Loft developed that file. The forum post a PHP script to read the contents of the dayfile.txt is a good starting point, it is about the development of ImportCumulusFile.
That post explains the date delimiter, how a Comma Separated Variable file may use other punctuation to separate the fields, how to specify the location of your dayfile.txt log to the script and the reason the key variable is used.
The thread contains several insights into how Cumulus was developed and it is useful to read Steve's posts in that thread if you are developing your own scripts.
If, having developed a script, you want to share it with others, publish it on the Cumulus Support Forum in the Web general sub forum. However, be prepared to deal with ongoing support, writing a script that will work out of the box for other people is impossible, there are so many ways that people set up their website. For example if your script accesses a database, be aware that not everyone has the same approach to missing data. Decide whether your script will only work with perfect data, whether you need to treat zeroes as missing data, or whether your script expects missing data to have "Null" in the relevant column. As MX is developed and supports a greater variety of sensors, you cannot assume that everybody uses the column names invented by David Jamieson and used by Cumulus MX, some people may want their database table to use daily derivatives that do not come from a field in dayfile.txt, a few might want to implement a more consistent way of naming columns. If your script uses jQuery, or PHP, be aware that the versions you have done your development on may not match the versions used by other people. Generally anything written in just HTML and JavaScript (despite evolving ECMA 262 standard) will be portable, everyone now uses HTML 5, CSS 3, and even Microsoft browsers use JavaScript, not their own jSCript (used by early Internet Explorer versions).
At the end of this Wiki page are several sections dealing with issues found with the earliest script that was developed, even the original author ditched that script, and adopted one written by another author. That said the example sites section shows that at least one site is still using that script, at a later version.
Key differences between the legacy Cumulus 1 and MX
Cumulus 1 only reads "dayfile.txt" when you choose to display a period, or edit the file. MX reads the entire file when it starts and uses the file for various charting operations including history chart drawing. Thus with MX it is more important to take a copy of the file once a day and use that copy for all the scripts discussed on this Wiki page.
MX has (optional) functionality to create/update a database table, that by default uses columns that are all derived from fields in dayfile.txt; if you choose the defaults for this table creation then another option will update the table if you use the editor on dayfile.txt. (You can choose to give the table a different name or different columns, MX provides Custom options to automatically upload (insert rows,replace rows, update particular columns) at various time intervals for a table with different columns, but it won't do this when you edit the file if either table name or table columns have changed).
Cumulus 1 does not include any functionality for creating/updating a database table to hold daily summarised data. You will find below various suggestions about how to get a copy of your file to your web server, and how to update a daily based table on your database server
Example sites
Do be aware that references to external sites either in the Wiki or the Forum may quote links that no longer work, the destination site might have closed down, changed its name, or just eliminated its past content in a revamp.
If you do want access to a script, or demonstration, that you can no longer find, post in the forum as it might still exist in an archive somewhere, or another Cumulus user may have a copy to share with you.
Third-party contributions
Here are some URLs where some of the scripts in the list earlier can be viewed.
- This Wiki page was inspired by the David Jamieson (DAJ) code. Its problems are discussed later on this Wiki page, and nobody uses it now.
- An improved version of that was developed by BCJKiwi, and a working example of that Annual Weather Data Summary Ver 1.2. November 2012 Script (PHP script) can still be found here
- A different carrier page (see below) is the Weather Blue set of Alternative Templates available from the French only site Météo du Québec and the international site Weather byYou. The link that used to be here for a daily summary page has been removed as that site no longer uses it.
- David's site https://www.fordingbridgeweather.co.uk/datasummary.php, now uses an improved version of the 'SQL dayfile data Viewer' script .
- There are many websites using a derivative of the YADR developed by beteljuice, including https://weather.wilmslowastro.com/datasummary.php and by selecting Reports, then YADR at https://meteo-wagenborgen.nl
Please update or add to these links if you find other exampless.
Getting a copy of dayfile.txt onto web server
If you are using Cumulus MX, the "Extra Web Files" settings page allows you to copy or file transfer at end of day. Set the local file to "data/dayfile.txt" (or if using Microsoft operating system "data\dayfile.txt"), remote file to destination, tick EOD column.
If you are using legacy Cumulus 1, use the method described here or Cumulus Toolbox to get a copy of the local file to use on their web server. (If your web server is local, you can simply copy the file to that server, once a day).
Populating a database table
- The ImportCumulusFile page describes a method that can be used with Cumulus 1 to mimic the contents of dayfile.txt in a database table. However, be aware that the later versions of that script have been edited for MX, so you will need to use an older version of that script that matches the version of Cumulus 1 you are using.
-
- CumulusMX.exe has an optional feature with a standard option to insert a new row into a database table at end of each day - see MX_Administrative_Interface#Standard_Daily_Summary_Table section for details of how database table holding columns relating to the dayfile.txt fields.
- ExportToMySQL.exe (see Software#ExportToMySQL), can update the database table with past rows.
Technical information
Displaying daily statistics from dayfile.txt
If you choose a script that parses a copy of the Dayfile.txt, then the script you choose might require some customisation:
- to define where the file is to be found,
- to define what character separates fields in the file, it does not have to be Comma Separated Values
- to define what character separates the day of month from month and from year
Some scripts might work out at least some of these from examining the file.
Displaying daily statistics from a database table
Cumulus MX has a standard name and standard set of columns it uses for the database table with one row for each day. If you have accepted these defaults, skip the next section, as it is likely that you are using a routine that knows the defaults and does not need any customisation for running against your database server.
If you use Cumulus 1, or you have created a database table with a different table name or with columns that suit you, then it is likely you have some database knowledge and skills, but the next section goes into some technical detail that might help you, although novices might want to skip it!
Schema
- The retrieval script needs to have the following defined for it to use (it is suggested that these variables are defined in a separate PHP script that is in a directory that the PHP parser can access, but is not in the part of the web server that is visible to a client):
- The server host name, or w.x.y.z format Internet Protocol (=IP) address, running your MySQL database
- The user name used to log-in to your database server
- The password used to log-in to your database server
- (It is good practice to have separate authentication for users with privileges of read only access, and for users with privileges of read and update access, alternatively use an additional key variable as David does in his update script described below)
- The name of the (MySQL or whatever) database with the table
- The name for the daily summary table
- The retrieval script needs to know the names of the columns it is to read from the database.
- Database designers talk about the schema meaning the specification that describes the design of the table.
- A schema lists the column names together with what type and size of variables they hold.
- For the daily summary database table, the columns can copy the fields in the daily summary log file, and this is what David Jamieson does in his approach a PHP script to read the contents of the dayfile.txt where the columns are generated in the same order as the corresponding fields in the log file, so David's script uses the field numbers to identify the columns by Position (rather than by Column Name).
- Remember that while the fields in the log appear in the order that statistic was added during Cumulus 1 development, in theory the columns can appear in any order (if all your scripts identify the columns by name rather than position) so columns associated with rain (these are separated in dayfile.txt fields 12, 13, 14, 31, and 32) can appear together, as can those for wind (these are dispersed in dayfile.txt fields 1, 2, 3, 16, 17, 18, and 39).
- If you have the skills to write your own script, then you might want to consider a different schema, daily information is in three log files for Cumulus 1, "dayfile.txt" should have the statistics for all completed days (that excludes today), "today.ini" has values for today-so-far, and the Weather Diary where snow records (and general weather text) can be manually recorded. A schema that combines information extracted from those three files to show everything the legacy Cumulus has for a day is available in this forum post.
Alternative schemas
Of course, the schema in your database table does not need to exactly mimic the fields in the dayfile.txt file.
- Perhaps you do not use all of the fields; your weather station may not produce solar values so those fields in dayfile.txt need not be columns in your database table
- Perhaps you want columns representing values from extra sensors or other log files.
- Functionality is included in Cumulus MX to handle your own schema (your choice of columns); for full details of the functionality see Cumulus MX Announcements - PLEASE READ THESE!.
Validation
The standard functionality in MX, and that described in ImportCumulusFile, assume that your data within Cumulus is perfect, you never have rogue values, nor do you have any missing data, and that you are using a weather station that has all the sensors to populate all the fields in dayfile.txt.
Thus your database table gets an exact copy of what numbers were found for all values and time-stamps in the logging file. Each of your scripts driving web pages for viewing data from your database table needs to include some validation to ensure that what is shown makes sense.
Alternatively, you might feel that you want to validate all the fields read from the daily summary file, before writing them to your database table to ensure your database only contains good quality data. Equally, to simplify querying your database table, you might want missing data to be stored as real nulls in the database columns, rather than the "zero" or "space" or empty character string or even sometimes 99.99 or -99, that Cumulus MX uses in its inconsistent way. That way you don't get any longer period averages skewed by the invalid numbers.
To implement those changes, you need a little knowledge about Structured Query Language, and you will need to write your own script (possibly based on the script in ImportCumulusFile, that validates what it reads from each field, and stores 'NULL' for any unacceptable content. Having ensured your database table only has good data, all your retrieval queries can much more easily ensure that you don't get what is shown on your web pages skewed by invalid data.
The Inspiring Script
The rest of this Wiki page can be skipped, unless you have a thirst for knowledge about the development of early scripts for processing this daily information. The text is retained purely as historic record, but if you are developing your own script you might want to understand what problems were hit earlier. For PHP scripts a lot of the issues faced by authors of scripts derive from the way that new releases of PHP add new features and deprecate others, this leads to better functionality but it can stop old scripts working!
Issues with DAJ script
The technical issues and the usability issues below, have been addressed by BCJKiwi's replacement script, by the SFWS alternative script, and by the beteljuice "Yet Another Dayfile Reader".
The following sections describe how DAJ's script made configuration complex.
Localisation / Language in DAJ script
DAJ claimed that his script had been designed to be easily translated to your language of choice.
Just edit the readDayfile.php
- variable mn is a list of the 12 months of the year, in an abbreviated format. You may change these as necessary, but try to keep it to an abbreviation as there is limited space.
- 'label_items' is a list of all possible data sets to be displayed. Again you can change these to suit if you are prepared to wade through all the script and make changes elsewhere too. However, the format is a little more involved and you should take some care. Each row represents one data set, with 4 columns of settings for that data set; so the default list is an array of four by six.
Example of a row,....
['maxtemp','Max Temp','Maximum Temperature',true]
- The first element, in this case 'maxtemp' -- is a system variable to identify the data set, used for the URL parameter option, used by the Switch coding (so any change here needs to be reflected there), and also tested elsewhere in the code (in the Rainfall and Windgust data sets a particular style is applied to their zero values).
Adding/removing data sets in DAJ script
If you add a new data set, then your new first element should be a unique identifier.
- Second, Max Temp is the text to be displayed in the button at the top of the table. You may change this to your own language
If you add a new data set, remember to consider how many buttons can be fitted across the top of the table, you may wish to split into multiple rows.
- Third, Maximum Temperature is the text shown at the top of the table to describe the current data set; again you may change this
- Finally, 'true' will display this button at the top; 'false' will hide it. Therefore, if you do not wish to allow users to jump to the 'Rainfall data set' change the 'true' to 'false' in the 'rainfall' element of the variable. (true/false MUST be lowercase)
Having made any change to the array as DAJ says above, you also need to change what is in a switch section and you will find that is below where DAJ says '// Nothing to edit below here'.
The switch ($whatdata) { ... } part of the coding identifies the columns in dayfile.txt for the values to be displayed. Any change you make in the array, needs a corresponding change within this part of the script. A typical case here reads
case 'mintemp' : $tablelayout .= $label_items[1][2]; $dayfilecol=4; break;
so to add a new one, copy that line and paste it in again but in this new line replace 'mintemp' with whatever you have added as the first element in your new array row, replace '[1]' with square brackets round the number of your new array row (the first row is number zero) and finally replace '4' with the field number where the parameter you have added is found in dayfile.txt.
Update of DAJ PHP script by BCJKiwi
There is a revised set of files (PHP only) ver 1.2. contributed by BCJKiwi on 17 December 2012 that update David's Ver 1.2 (November 2012) fileset to eliminate errors in validation due to non-standard or deprecated code routines. They also include a couple of updates from DAJ version as advised by posts in the Forum. The text that follows describes issues with David's script, and how they were rectified by Brian's script
See AnnualDataSummary page.
Configuration Variables
One issue with early scripts was that the user had to edit the files in the package to configure the script to work in their installation, here are details of the sort of issues end users had with the assumptions in those scripts.
By default in 'readDayfile.php':
- Assume the dayfile.txt is read from a subfolder called 'data'.
- Look for the line 'dayfile='/data/dayfile.txt';' around line 15 and change this to point to your dayfile.txt on your webserver.
- for PHP version the path should start from the / (root)
- Assume the date format in the dayfile is dd/mm/yy
- You may need to edit the file if you do not use '/' between the parts of the date (the month is always in the middle for all Cumulus log files), look in file for date_delimiter' - the symbol separating your date format. See setup.
- Assume the data in the dayfile is separated with a comma
- Look for the line mentioning field_delimiter - the symbol separating each of your fields in the dayfile.txt. For most people this is a comma but (if you use comma to separate integer and decimal parts of real numbers) it could be a semi-colon (;) or other symbol.
Errors rectified by Brian's version
- Non-standard attributes 'datacol' and 'datarow' are generated by the original javascript and php routines that will not pass validation against standards (in HTML5 these can be made to validate by changing to 'data-col' and 'data-row').
- In version 1.1 of David's script, jQuery version 1.5 is called, this permitted use of '.attr' object to add/remove the highlight;
- this will not work for later jQuery version 1.6 asuse of that object was deprecated and the '.prop' object is now specified for the same purpose.
- Version 1.0 and 1.1 of David's script default to showing a table for viewer's local current calendar year. Remember the dayfile.txt (even if updated each day) only contains records up to the day ending at the last rollover time using the timezone local to the weather station. Timezone differences when combined with rollover time variations can lead to a period of 2 days discrepancy. At New Year a blank table will be displayed during this period by default. For example, 30 or 31 (depending on whether rollover has occurred) in December of the last year could be the latest available for a site in USA whilst the original version of the code is showing a blank table of the next year during early morning on 2 January in Australia. Even when station and viewer are in same timezone, there will be a blank table until the first rollover on 2 January.
- Version 1.2 of David's script defaults to showing the year found in the last row of dayfile.txt.
- The final field on each row is not read correctly, because the row break has been specified wrongly for a Microsoft Windows environment, it should quote fdata.split("\r\n"); i.e. defines that carriage return then line feed (newline) used to split daily observations in dayfile. Because the script presented here only processes a small number of fields from earlier in the row, this error does not affect the output for the published code.
- See more background in forum thread AnnualDataSummary PHP.
Instructions for using any of these scripts
The instructions on David's original Wiki page for PHP version still apply to most subsequent alternatives, so these are repeated below in a simplified manner. The files in the zip include additional notes, and explanations on the changes made in the update, where relevant.
- Download the following zip file...AnnualDataSummary_12.zip, the zip contains 3 files:
- readDayfile.php - script for reading the dayfile.txt and for creating the table structure that displays selected statistics in a calendar style
- datasummary.php - example 'carrier' web page, this one uses the Saratoga template system, so also requires 2 more scripts (not provided) Settings.php and common.php. You can use your own page instead of this file - see carrier page customisation below.
- datasummary.css - styling sheet for the table structure that displays selected statistics in a calendar style
- Edit the file 'readDayfile.php' in a good text editor (for example, notepad++) and consider the configuration variables described above that appear in the top section of the script.
- After any necessary editing, copy datasummary.php and readDayfile.php files to your web site into a subfolder, or the root -- your choice
- Copy datasummary.css (for possible tailoring see styling instructions below) to your web site into the same folder
Usage
- Open datasummary.php from your website in a browser
Along the top will be a menu of the (in original version) six data sets available and on the top left of the table is the year currently being shown. Change either the year, or click one of the top buttons to change the data set. Depending on the speed of your internet connection, and your browser, the new web page may take a few seconds to be processed as it involves: re-reading the dayfile.txt file, redrawing the table, and inserting the values.
If you mouse over a particular value, it will be highlighted together with the corresponding day (row heading) and month (column heading).
Opening a specific data set when the page is first loaded
By default (in the original version) the table created will show the available daily values of the maximum temperature for the current local calendar year, however you can start with any data set and any year (assuming you have those values in the dayfile.txt) by adding a 'Query String' to the end of your URL in the browser.
- The parameter 'data' (in the original version) takes mintemp, maxtemp, avgtemp, minmaxt, rainfall, windgust. (You can add further data sets or change the language of these values - see Localization). Note the specified data will be displayed even if it is assigned 'false' and does not appear as a button.
- You can supply both data set and year parameters, one only, or none
examples....,
- myserver.com/datasummary.php?year=2010
- This will open the default (maximum temperature) data set but initially showing year 2010
- myserver.com/datasummary.php?data=rainfall&year=2009
- This will open the rainfall data set for the year 2009.
Understanding the Basics
The original 'tool' relies on three files for successful operation:
- the jQuery library routines that allow it to work in any user agent (browser)
- datasummary.css (the style sheet) and
- readDayfile.php (the PHP version of the script).
Please note that the table needs a good amount of space to show a full year of data (at least 900 pixels unless you start reducing the font size!)
In addition you need a carrier page to show the data.
- Note that BCJKiwi does not use a web page based on those that are supplied with Cumulus, as mentioned above his provided datasummary.php requires two extra scripts not provided. If you create two empty files with the specified names, or download files with those names from Ken True's web site, then the provided carrier page will load on your site.
- Alternatively don't bother to upload the provided datasummary.php, and instead create your own carrier page (use the same name or call it whatever suits you) following instructions below.
Creating your own Carrier Page
To create your own page, follow another page you are using successfully on your site and modify it by removing any content from <body> section you don't want and including in the <head> section:
- the CSS styling:
- (HTML best practice is to include CSS declations as early as possible).....
<link rel="stylesheet" type="text/css" media="screen" href="datasummary.css" />
- the script library (this example shows version 3.1, you can load whatever is the most recent version suitable for your browser):
- This loads a jQuery library from Google servers rather than having the file on your server, (note many versions of jQuery are not supported by old versions of Internet Explorer, use the very old version '1.5' used by DAJ instead of the current version where '3.1' appears if you need to);
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.1/jquery.js" type="text/javascript">
- Next you must load the script to build the table:
- In the <body> section, use the code
<?php include('readDayfile.php');?>
(or simply copy everything that was in that file and paste it in this section directly) to include the script instructions that create all the HTML elements required to build and display the table
Styling
The table styling is completely configurable using the included Cascading Style Sheet datasummary.css. By default it is using similar colours to the standard Cumulus website. Below are a few of the key entries to consider when adjusting settings (typically colouring)... Also see support forum thread about colouring with php
In the 'datasummary.css' file:
CSS Element | Description |
---|---|
#table_container .highlight | Used to highlight the mouse position within the table, and also the row and column header of the data cell. |
#table_container .smallfont | Adjusting the font size to something smaller when showing both Max & Min temp on one data cell |
#table_container .zerovalue | In the Rainfall and Windgust data sets this style is applied to any values of 0. By default, the colour is set to a lighter grey but you could add 'display:none;' to hide zero values completely |
#table_container table th | Colouring for the top header (Month names) and left header (day numbers) |
#table_container table td,table th | The width of each data cell in the table. Be careful adjusting this as making it too small will stop the data being displayed completely |
#table_menu li | The styling for the buttons at the top of the page (those for changing the data set) |
#table_container | Set the overall font size and style used in the table, as well as the text colour; table positioning and maximum width |