Monthly log files: Difference between revisions
From Cumulus Wiki
Jump to navigationJump to search
m (→Potential Name Confusion: rewrite of this section) |
|||
| (31 intermediate revisions by the same user not shown) | |||
| Line 1: | Line 1: | ||
[[Category:Cumulus Files]]Please see [[Standard log files]], [[Extra Sensor Files]], [[Air_Link_Log.txt]], [[Month.ini]], or [[Monthlyalltime.ini]], as relevant. |
|||
<big>The Standard Logging File in Cumulus</big> |
|||
Please be aware that all of the above links lead to documentation about files that may be described as ''monthly log files'' when reading individual Cumulus forum and wiki references. |
|||
= Introduction = |
|||
==Potential Name Confusion== |
|||
This article is about the standard logging file that stores the full set of current (read and derived) values at whatever interval you have set for Cumulus to keep detailed logging. The default time between data logging lines is every 10 minutes, but you can change this: |
|||
*Cumulus 1: From Configuration menu, select the [[Cumulus_Screenshots#Station|station settings screen]], '''Data log interval''' box on extreme RH side. |
|||
*Cumulus MX: From settings tab, select the '''Station settings''' screen, and find '''Log interval''' after the ''Ecowitt GW1000 Settings'' section. |
|||
Those who used early versions of Cumulus will remember that the term '''Log File''' was originally used for [[Log.xml]], but Steve Loft later accepted that all the files with a [[:Category:Ini Files|.ini]] file extension could be described as log files, so he introduced the term [[Weather Diary]]. A bit later, [[Standard log files| the detailed data file]], began to be also called '''log files''', and the confusion grew. When [[Extra Sensor Files]] were added, these also became '''log files'''. The Wiki page now called "Standard Log Files" was first called "Log files", then became "Monthly Log Files" because "Log files" became a category. Then "Extra Sensor Files" were added and "standard log files" replaced "Monthly Log Files" to make it clearer which file held the standard values, and which file held values for extra sensors. |
|||
These files are automatically created with names in the form <Month><Year>log.txt (for example, the file for August 2009 is often called <tt>Aug09log.txt</tt>), but the month can be in another language or a different abbreviation depending on your locale. There is no setting to change this format. |
|||
The standard logging file is in comma-separated format with one line per entry. Being comma separated does not mean the fields in this log are always separated by commas, they might use semi-colon (;) for example. |
|||
Steve Loft wrote the Cumulus software mainly for his own use, although after a while he let others use it and added features that others requested. Cumulus was developed in his spare time outside a full-time employment, and consistency and terminology were not issues he worried about. |
|||
There is an equivalent logging file for [[Extra_Sensor_Files|extra sensor]] values. In fact, in Cumulus MX, the processing of this extra sensor logging file shares a lot of the code with the processing of the standard logging file, as they both are updated at same interval, and they both take values processed following a read from your weather station. The applications programming interface (api) that sends values between the MX engine and the MX admin interface is also identical (with a parameter "Extra" that swaps between true and false depending on the log file involved). |
|||
During the development of the original Cumulus software (Cumulus 1 and 2) by Steve Loft, the name "monthly log" was most frequently used as a collective term for the last two (.ini) files linked above because those files tracked the monthly extreme records. |
|||
There is another logging file, [[speciallog.txt]], for detailed logging of internal temperature and humidity, but it is not split into separate files for each month. Although this exists in Cumulus 1, it appears this functionality is missing from MX. |
|||
When Steve Loft was working on his [[Cumulus_3_(MX)_beta_documentation|MX Beta]], he decided to implement within MX functionality the database schema that Mark Crossley had designed on [[ImportCumulusFile#Monthly_table_.28old.29|this]] page, where Mark selected the name '''Monthly''' for the table. David A Jamieson created versions 1.0 and 1.1 of ImportCumulusFile for creating a single table called '''dayfile''', but Mark extended it to cover ''realtime'' and what he called ''monthly log files''. That means the standard log can in MX optionally feed data into a database table called '''Monthly''' by default in MX. |
|||
{{TOCright}} |
|||
This database table terminology can be misleading as ''that database table is not organised by month in any way'', a single table contains every logged line that has been inserted into it by Steve's '''ExportMySQL.exe''', Mark's '''ExportToMySQL.exe''', or by '''CumulusMX.exe''' itself. In fact MX allows you to give this table another name, which it will then use in all SQL. |
|||
= File content = |
|||
Mark Crossley has taken over development of MX, the terminology "monthly log" is now being used for the first linked file (standard logs) instead of for '''month.ini''' the file that is truly logging monthly extremes. The name has been adopted simply because a new log file is started for each month. |
|||
This file has one line added each time the Cumulus timer says the logging interval has passed. |
|||
For novices, there can be a feeling of confusion: |
|||
All data is logged in the units as selected by the user: |
|||
* The new practice leaves "month.ini" without a descriptive name |
|||
*Cumulus 1: From Configuration menu, select the [[Cumulus_Screenshots#Station|station settings screen]], '''Units''' box is just below the centre of that screen. |
|||
* Each "dayfile" line represents a new day so the filename matches the content, but a line in "Oct16log.txt" does not represent a month, and the file name does not contain the phrase "month" |
|||
*Cumulus MX: From settings tab, select the '''Station settings''' screen, and find '''Units''' section at the start of that screen. |
|||
* The names "Month.ini" and "Monthlyalltime.ini" do contain the phrase "month", they do contain monthly data in each line, and do log extreme records that clearly identify with a particular month |
|||
A future Wiki editor may decide on a different strategy, but at the moment the Cumulus Wiki has returned to the original Steve Loft terminology for each of the log files, for consistency with older articles in this Wiki, and the oldest posts on the forum. |
|||
=Enhancement never implemented= |
|||
Each line has a number of fields separated by whatever symbol is defined as the list separator on your device, typically a comma (,) or a semi-colon (;). |
|||
* The various fields are all listed later in this article, but they can be of various types: |
|||
** Date, this is always the first field, and it always uses a fixed format, ignoring the format defined in your locale. |
|||
** Time, this always the second field, and is always in the form hh:mm (Cumulus will not be able to understand the file if you edit the time to a format that includes seconds) using the 24 hour clock and local time (system time). However, if you use the feature in MX to upload the equivalent of this file to a database table, the upload will add ":00" to include seconds. Note that it is perfectly possible to have two successive lines in a monthly log file with same time, although this should not happen in normal running, it can result from a hiccup or other interference on your device. If you are asking MX to upload the contents of this file, a second line with same time is ignored, only the first line with a particular date and time is uploaded. |
|||
** All Remaining fields are spot values, either what has just been read from your station, or a derived value calculated from a set of values read from your station. For these value fields, Cumulus may have converted from the unit used by your weather station to the unit that you have requested to use, and there will be some inaccuracy inevitable in such conversions. Where the value to be stored is a real number with decimal places, Cumulus will use a decimal comma or decimal point depending on which is defined in your locale, do be careful not to change that locale if you need to change your device or update it, as Cumulus can only read files where the same decimal separator is used everywhere. |
|||
Steve Loft was several times asked if he could change the design of his software and what it stored in [[Standard log files]], [[Extra Sensor Files]] and the other detailed logging files. |
|||
All these files currently periodically store spot readings. Some of the competition in weather software, periodically store the highest and lowest since the previous line was stored. |
|||
For ''more information'' on these files see in the '''Cumulus help''' file, in the section “The Data log file”. This is installed with Cumulus 1, but can be read with Windows after downloading from [[Software#Resources]] page in this Wiki. |
|||
One advantage of the alternative approach is that you never lose any extremes. Another advantage is the alternative allows people to report extremes for parts of days. For example, some people wanted 9 am to 9 pm reported separately to night time extremes, in other nations, the dividing time was 3 pm not 9 pm. |
|||
= Accessing the monthly log file = |
|||
Steve Loft would respond he had made many design decisions that were not ideal, but the software was based on his own needs. If people wanted drastic changes, they should do what he had done, and write software that did what suited them. |
|||
== File Locking restricting access by other processes == |
|||
Yet it appears he did plan to implement this suggestion in MX, see [[Cumulusmx.db#Releases 3.0.0 to 3.11.4 inclusive]]. |
|||
As the log file for the current month will be updated frequently by Cumulus (when it is running), Cumulus applies an exclusive lock, and (as explained further below) conflicts can happen if another process seeks to access this file. Consequently don't let your antivirus scans access this file, nor try to edit it outside Cumulus while Cumulus is running. |
|||
Also note these log files do not include a header line, and should not be edited to include it. |
|||
The appropriate header file for the monthly data logs can be found for any installed version of Cumulus by looking for "monthlyfileheader.txt", in the directory above the 'data' directory containing the monthly log. The viewing methods for the two flavours, as described next, automatically includes the header line. |
|||
A full discussion of the problems with conflicts of access to the monthly log file can be found [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721 in this support forum topic]. |
|||
== Cumulus 1 == |
|||
In the '''View''' menu of Cumulus 1, select ''Data logs'', click '''Load...'''. |
|||
The standard Windows File Select dialogue is displayed for the 'data' subdirectory of the Cumulus installation directory. |
|||
Select the required monthly log, click '''Open''' and you will see the contents in a neat table with column headings, rows striped, and the date/time on a lighter background. This is a text viewer, and works best when at full screen, but even then you are likely to need to scroll both horizontally and vertically to look at all the figures. If you click on an individual figure the cell will be highlighted, but you cannot edit the figures on a view screen. Cumulus 1 does not provide an editing facility for this file. |
|||
=== Problems === |
|||
If no logs seen, see [[FAQ#I_can.E2.80.99t_find_my_data_files.21|FAQ: I can't find my data files]]. Also note, this '''Data logs''' viewer is only for viewing standard log files (not any other files in the same ''data'' directory), the column headings for standard (monthly) log files will always be shown at the top of the screen. |
|||
== Cumulus MX == |
|||
In the admin interface, there is a "Data Logs" tab, from it select ''Data logs''. |
|||
*On this screen,you see a box for selecting the log you want to display. The default month and year (shown on loading the web page) is taken from yesterday's date. No log is loaded at start up. You can '''either''' type in a period (one or two digit representing month, then hyphen, then four digits representing year) '''or''' select year then month in the data picker (this is produced by the bootstrap software MX uses, it is not the date picker provided by your browser), when you click on the month the box is updated with selected period. |
|||
* Next to it is a "Load" button. Press it and the Cumulus MX engine will generate an application programming interface (api) table that will appear on the web page using datatables software. There is a "Refresh" button, for when you want that api to refresh the web page (without reloading the whole page). The table generated is paginated, it shows a fixed number of lines, and there are arrow icons to move to next, and back to previous page. |
|||
When you click on any single line, two extra buttons (for editing that line and for deleting that line) are enabled. The Delete a line button brings up what the datatables software calls a modal, a pop up dialog where you see the fields on that row listed vertically and have two more buttons (Cancel and Delete) with obvious consequences. |
|||
How to edit the log using this functionality has changed already for different MX versions and is planned to change again, so I am not describing it further for now. |
|||
=== Problems === |
|||
If you try to look at the current log, and it does not display in the MX admin interface, it might be because you are trying to view it at the same time as the MX engine is writing the next line to it. The exclusive locking stops the admin interface and the engine being able to both access the same file at the same time. So if your logging interval is every ten minutes, don't try to view it within 2 minutes of the hour and within 2 minutes of every ten minutes thereafter. That still gives you 6 minutes between each 10 minute update for viewing! Of course there is no conflict for past monthly logs. |
|||
== Additional ideas for how Monthly log files can be used within Cumulus == |
|||
=== Uploading current log to your web site === |
|||
The current monthly log can be updated at your usual web site update interval using ''<currentlogfile>'' as the local file name. If your web site upload interval is the same as the logging interval, there might be a conflict because of the file locking, so this might not work every time. However, extra web files are not processed all at once, so by listing this file against a higher number the chance of conflict is reduced. |
|||
For Cumulus 1, list this log file as the local file by specifying the parameter 'ExtraLocal[0-99]=<currentlogfile>' in [[Cumulus.ini#Section:_FTP_Site]]). For Cumulus MX, use the '''Extra web files''' option within the "Settings" tab, and enter "<currentlogfile>" in the local column. |
|||
You can also use <currentlogfile> as part of the remote file path, as explained in the Cumulus '''Help'''. You will also need to specify a path in the remote column for MX or the 'ExtraRemote[0 to 99]=...' for Cumulus 1. You might of course want to specify a local path, and leave the FTP column unticked (or set to 0 for Cumulus 1). You can then use a separate FTP process to upload this copy. It is not recommended that you do not try to FTP the actual log file (only a copy) because the FTP process is relatively slow, particularly as the file size increases as the month progresses, so there is an increasing chance of conflict with logging - see [https://cumulus.hosiene.co.uk/viewtopic.php?f=4&t=12721&p=99793#p9979 support forum here] for more information. An alternative approach is to upload the monthly file just once a day, at a quiet period such as an odd number of seconds after quarter-past midnight using a process scheduled outside Cumulus. |
|||
=== Cumulus 1: transferring the contents of the monthly log file into a database table === |
|||
A script can be used to read the your copy of the monthly log file into either a database table (see [[ImportCumulusFile]]), or into an array, for further processing. It is, as advised above, best not to work with the file that Cumulus is actually updating, because there can be a conflict of the locks, Cumulus needs to have exclusive use when it does its normal update. But this script can safely be used with older monthly log files, or a copy of the latest monthly log file. |
|||
Although the '''ImportCumulusFile''' script can be used with MX too, MX provides 2 other ways to transfer the contents of a monthly log file into a database table. The first, uploads past data, using '''ExportMySQL.exe''' found in the top level folder where ''CumulusMX.exe'' is found. It can be run using the same way as you run the main program, but you need to specify "monthly" as the first parameter. The second, which only uploads the line most recently added to the monthly log file is found in the '''My SQL''' settings section of the admin interface. Find more information at [[Cumulus_MX#MySQL_settings|MySQL settings]]. |
|||
== Editing the log file == |
|||
=== Editing in Cumulus MX === |
|||
This changes between versions and the text that was here has been removed. |
|||
=== Editing for Cumulus 1 users === |
|||
Cumulus 1 allows you to view any monthly log file, but not to edit the contents. Consequently for Cumulus 1, you need to edit monthly log files outside Cumulus 1, subject to the following rules: |
|||
* You can't edit the current log file while Cumulus 1 is running, because Cumulus needs exclusive write access. You can edit the log file for a past month, unless you are looking at past months using Cumulus editors. |
|||
* You can't edit any log file with a word processor, as they add control characters and other information that Cumulus cannot understand. |
|||
* You can use either a specialised comma separated value file editor or a text editor. |
|||
* Text editors designed for programmers will allow you to select the encoding (Cumulus will be confused by any Byte Order Mark, so select the encoding type without BOM). |
|||
* If you choose to use a spreadsheet, ensure that all columns are treated as normal text, do not let (don't accept Excel default) the spreadsheet recognise the first field contains a date as it will convert that column into a number (e.g. days since 1900 or days since 1970). For example in Libre Office make sure that "Detect special numbers" is not selected. Many spreadsheets will offer a CSV option for saving the file (in Libre Office tick "Edit Filter Settings" on "save as ..."). |
|||
* If you amend a field, ensure that replacement is same format as original (same decimal separator if not integer). |
|||
* Ensure no blank lines, and if you have added in the column headings then remove that heading line when saving ready for Cumulus. |
|||
You can see [[#Manipulation outside Cumulus]] below for more information. |
|||
===After you have edited (or created any missing) Monthly log files, you can:=== |
|||
* HOW TO CORRECT HIGHS AND LOWS |
|||
** update the highs and lows in [[Alltime.ini]] by choosing ''all time records'' from the '''Edit''' menu. See [[Alltimelog.txt]] for current and previous values. |
|||
** update (if the created/edited monthly log is within the current year) the highs and lows in [[year.ini]] by choosing ''This year's records'' from the '''Edit''' menu. |
|||
** update (if the created/edited monthly log is for the current month) the highs and lows in [[month.ini]] by choosing ''This month's records'' from the '''Edit''' menu. See [[Diags]] for current and previous values. |
|||
** update (from version 1.9.3) the highs and lows in [[monthlyalltime.ini]] by choosing ''Monthly records'' from the '''Edit''' menu. Click the ''Help'' button for specific instructions on using ''Reset'' and the two ''Copy'' column header buttons in this ''Monthly Records (Highs and Lows) Editor'' to action all rows. |
|||
'' '''Note''' in each of above 4 editing screens you can: |
|||
# see the currently stored extremes, and optionally ''Reset'' (row by row) to pre-editing value and timestamp. |
|||
# load the monthly log(s) and dayfile.txt to view extremes calculated from those figures (as available) and |
|||
# optionally (in Cumulus 1 only) click ''Copy'' (row by row) to move the logged values (and associated date/time information) into the relevant .ini file. In the MX editor, you need to manually type the numbers read from the other logs into the monthly log column. |
|||
# (In cumulus 1 only) click the ''Help'' button for detailed instructions on using ''The Records (Highs and Lows) Editors''. |
|||
# In cumulus 1, store your revised figures by clicking ''OK'' (or abandon all your edits by clicking ''Cancel''). In MX there are tick and cross options for same effect. |
|||
(Each of these screens is a text editor, and works best when at full screen).'' |
|||
*HOW TO USE CORRECTED INFORMATION |
|||
** (In cumulus 1 only) see '''[[Average_temperature | Mean temperatures]]''', '''[[Heat/cold_degree_days_and_Chill_hours | Air frosts]]''', '''[[Gale_Days |Gale days]]''', '''[[Rain_measurement#Precipitation_measurement_and_Cumulus | Rain days, Dry days]]''', and '''Total [[Windrun |Wind run]]''' using '''View''' menu displays for ''This month'', ''This period'', and ''This year'' (These figures are not stored anywhere, nor available as [[Webtags|web tags]]), but Cumulus 1 can calculate these from what it finds in the monthly logs and the daily summary log. |
|||
** (In cumulus 1 only) use '''create missing''' button in the ''[[dayfile.txt]]'' selection in the '''Edit''' menu; This is a text editor, and works best when at full screen. Cumulus will then look through all monthly log files and create approximate records for any ''missing dates'' in the dayfile if those dates have observations stored in the relevant monthly log. (It creates a file in the Cumulus folder called ''dayfileeditlog.txt'' which contains the entries it created). Click the ''Help'' button for more information. In past and current versions (including 1.9.3), ''create missing'' will not affect any records that are incomplete or contain some rogue values (see [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] below). |
|||
** obtain (in Cumulus 1 by [[Monthly_log_files#Viewing_monthly_log_files | viewing the created/edited log file]]) figures needed for manually correcting rogue figures in [[dayfile.txt]]. See for example [[FAQ#My_monthly_and.2For_annual_rainfall_totals_are_wrong | FAQ re correcting rainfall]]. |
|||
** create the relevant monthly and/or annual '''NOAA style report''' by choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. |
|||
***In Cumulus 1 do this from the '''View''' menu, then select the required period using the selectors. Click the ''Update Display'' button to see various statistics (including [[Average_temperature|mean temperature]]) calculated. Generation of complete NOAA reports takes most information from dayfile.txt (based on rollover to rollover meteorological days), except average wind speed and dominant wind direction (both of these it calculates from the monthly log files) for period in question. Finally press ''Save'' button to store the new or amended report. |
|||
***In Cumulus MX, use the '''Reports''' tab, then choosing ''NOAA Monthly Report'' or ''NOAA Annual Report''. A date selector allows you to choose the exact report you want, click "Load" button next to it. You can alternatively click another button to generate a missing report, or to regenerate an existing report after any corrections you have made to monthly log file. Be aware that a regenerated report might not be quite as accurate as original report which when generated is using figures actually being processed by end of day for the current day. |
|||
'''NOAA Report Notes:''' |
|||
* for obsolete versions up to 1.9.4 build 1085 only: The average wind speed used for NOAA reports and 'This period' type screens was, by a bug, based on midnight to midnight days regardless of rollover time in use. (It is calculated on generation of report or screen from monthly log entries). |
|||
*From build 1086, the calculation is based on the rollover time being used. |
|||
* For all Cumulus 1 versions, and the MX versions below 3.4.4: the yearly average temperature is calculated from summing monthly maximum and monthly minimum values and dividing by number of values in calculation. |
|||
* From version 3.4.4 the annual average temperature is calculated using the integrated method based on every temperature reading processed (it is actually the average calculated from all daily means stored in daily summary log for this year). |
|||
==Correcting any logged data problems== |
|||
As already mentioned, it is only MX where the 'Monthly Data Logs Viewer' screen will allow editing of values, but in both flavours you can use their viewer to look and see if you are happy with what has been recorded. |
|||
Some Weather stations may occasionally supply corrupted or rogue values to Cumulus. |
|||
*Cumulus 1 provides via '''Configuration''' menu [[Cumulus.ini#Section:_Offsets|''Calibration'']] screen the ability to screen out spikes (i.e. abnormal differences between two successive readings from weather station) in data picked up from some weather stations, it does not work for all types. See Cumulus help screen if you decide to use that to cope with future spikes. |
|||
*Cumulus MX provides the same functionality, for the same restricted set of station types in its admin interface. |
|||
Both flavours of Cumulus also store the same value for a short time, if a replacement value cannot be obtained from the weather station. They both store zero if no value can be obtained. |
|||
Some rogue values may be obviously invalid (e.g. a zero pressure), others (e.g. wind shake on a tipping bucket rain gauge on a fine day) may be within the accepted all-time range, but you know the value is not right for that particular time. |
|||
You may choose to just edit the rogue value into a reasonable value, or (if you are unable to decide on a better value) to delete that particular entire row in the monthly log. In Cumulus MX you can do both of these in the data logs screen, in Cumulus 1 you will need to use an external editor. |
|||
As explained above once any monthly log has been corrected, new highs and lows in other logs are easily rectified. |
|||
=== Spotting rogue values === |
|||
Frequently, the graphical views ('''Charts''', ''Graphs'' or ''Trends'') are the best way to spot many rogue values and to help you guess by interpolation a more reasonable value. |
|||
Another possibility is to examine the relevant values in previous versions of the log files stored in the ''backup'' or ''backup/daily'' subdirectories of where your cumulus executable is stored. |
|||
===Manipulation outside Cumulus=== |
|||
Any log files for ''previous'' months can be edited (outside Cumulus) with Cumulus running, and after editing them, you can do further fetch edits within Cumulus as described above. The current month log file can also be edited outside Cumulus, but as it is being updated frequently you must stop Cumulus first (that explains why there was no edit option within Cumulus 1!). |
|||
'''Tips''' -- take a copy of the original log file before you work on it outside Cumulus (perhaps give the ''not to be touched copy'' a filename of "<Month><Year>log.csv"). |
|||
Edit the original file using an editor that treats all fields as text [use either any text editor, a Comma Separated Value editor, or a spreadsheet program that can be instructed ''not'' to recognise special field (like date and time) types]. [[File:Open office (editing cumulus log files).png | left]] If you wish to use "Calc" in 'Apache Open Office', "Libre Office", or similar, select the field separator you use (in this illustration comma is selected, but your file might use semi-colons between fields, don't select commas if your real numbers use comma between integer and decimal parts) and leave "Detect Special Numbers" unselected (as normal default for this software). Best not to use 'Microsoft Excel' as that normally has opposite default, and may change without any warning the format of all dates so Cumulus can no longer recognise the first two fields: |
|||
'''Important Rules''' |
|||
*Each line must contain the fields in correct sequence (since new versions/builds can add to number of fields, Cumulus will accept different months having various row lengths i.e. older ones without the more recent fields at the end). |
|||
* The date format uses two digits for the year and should be treated as text. For software (e.g. Excel) with default of recognising formats, ensure that such recognition is turned off, as it is likely to change the dates to either a number representing days since e.g. 31 Dec 1899, or to have four figure years, and then Cumulus will no longer be able to use the log file. (Also the month must be the middle figure, USA convention cannot apply within ''Log_Files''). |
|||
* Times in these files are in the form ''hh:mm'' using the 24 hour clock and local time (system time). See [[FAQ#How_does_Cumulus_handle_Daylight_Saving_Time.3F]]. |
|||
*See [[#List_of_fields_in_the_file]] to identify if the value you enter for a particular field is an integer (like bearings and humidity values) or a real number (like pressure and rainfall values) in format x.y using your system decimal notation for the decimal point. Fields that accept real number format will also accept integers, but fields expecting integers will not accept decimal points. |
|||
*Negative values are allowed for temperatures, all others must be zero or positive. Humidity has a maximum of 99, bearing has a maximum of 360. |
|||
*Nulls (',,' if your field separator is a comma, ';;' if your field separator is a semicolon) are not allowed in the row, so if you do not know the value for a particular field within the row, then type in a zero, -999, or 9999 (check the format in the middle column of [[#List_of_fields_in_the_file]] and think about the range of numbers allowed for particular fields as per examples in previous bullet). In other words put in an obviously strange number as Cumulus does not have a defined null! This strange number may distort graphs and any calculations, but it will remind you to make a more appropriate guess as soon as you can. |
|||
* Note that some fields (e.g. those related to evaporation, UV, solar data) will only contain valid data if your station has the appropriate sensor(s) (without the sensor they contain the default value of zero), but if they apply make sure you note which are stored as integers. |
|||
===Importing pre-Cumulus data=== |
|||
See [[FAQ#I.E2.80.99ve_just_installed_Cumulus.2C_and_it_didn.E2.80.99t_download_all_the_old_data_from_my_weather_station|FAQ: I've just installed Cumulus and it didn't download all the old data from my weather station]] |
|||
Given that monthly log files are used as input for '''updating the all-time, month-by-month alltime (from version 1.9.3), and this year record extremes''', for '''creating missing dayfile.txt entries''', and for '''creating NOAA style reports''', you may have some observations from your weather station recorded manually or electronically for a period before you first starting using Cumulus and want to create new monthly logs to feed into Cumulus. |
|||
'''Here are some issues to consider.''' There are some postings in the Support Forum about importing past data. Essentially match the fields listed in [[Monthly_log_files#List_of_fields_in_the_file|Monthly log files: List of fields in the file]] below with the fields you have available in your source. |
|||
* Dates and times might need some pre-processing (spreadsheet packages usually have ability to select part of a text string and to cocatenate a number of strings) to convert them to text in the formats mentioned above. (Note - it is crucial in a spreadsheet to treat the date as text; Do not let the spreadsheet recognise it as a date as it might change the format or store as a 'days since X' number). |
|||
* Remember that null values are not allowed within a row, so ensure that you enter an obviously wrong value (-99 or zero might be do) between your operating system defined field separator (normally ',' or ';') and make the row length the minimum shown in the list of fields below. If you use a spreadsheet, treat the file as Comma Separated Value type, but on saving choose the appropriate field separator character (e.g in Libre Office or Open Office 'Save as CSV' select 'Edit Filter Settings'). |
|||
* Wind directions might be reported as cardinal points (north, south etc) and need converting to bearings in (integer) degrees (spreadsheet packages provide ''look up'' functionality to do such conversions). |
|||
* Wind speed fields may not directly match to weather station outputs. See [[#Cumulus_Wind_Speed_Terminology]] |
|||
* Temperature fields may not directly match to weather station outputs. Copy across the fields you can and set the others to -999.99 so they are obvious. See the [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] described below for a way of getting Cumulus to generate (from your newly created monthly logs) apparent temperature, wind chill and heat index to replace any -999.99 figures. |
|||
* Rainfall fields may not directly match to weather station outputs. [[FAQ#How_is_my_rain_rate_calculated.3F | Rain rate]] in field 8 may be generated from the [[Monthly_log_files#Using_Monthly_logs_to_deal_with_shorter_.28or_incomplete.29_dayfile.txt_records_for_particular_dates | workaround]] if unavailable from your station. The daily total in field 9 is since rollover time, and available weather station outputs may need some processing in your spreadsheet to derive this. Some stations output a count of total rainfall, others the total annual rainfall, either of these may be used for field 11. Rainfall processing issues are frequently discussed in the Support Forum, so you may find the solution there or ask there for assistance. |
|||
'''The date that Cumulus first started tracking all-time records''' (<#recordsbegandate> see [[Webtags#Records]]), '''does ''not'' need to be updated for Cumulus 1''' to recognise earlier monthly log files in its ''data'' subfolder and to use any monthly logs found in the various '''View''' and '''Edit''' options available from the Cumulus main screen. |
|||
'''Cumulus MX is different''', it only reads monthly log files from the existing ''records began date'', so that needs to be edited (''StartDate='' line in [[cumulus.ini]] with MX stopped, because it is not on any settings screen) to allow any earlier data to be read. You will also need to update all the other log files that record extremes manually using the editors provided in MX. |
|||
As <#recordsbegandate> is on the default Cumulus ''recordsT.htm'' template with the label "Records began on", inserting pre-Cumulus data implies you might want to amend that label wording on that template to indicate it is the date that Cumulus was first operational, rather than the date of the earliest tracked highs and lows. |
|||
If you prefer to edit the start date and keep the ''recordsT.htm'' template label "Records began on" unchanged, the method is to stop Cumulus 1 or MX, and amend the ''StartDate='' line in [[cumulus.ini]] within the main Cumulus folder before re-starting Cumulus, but (in case you make a mistake) back up everything first! |
|||
=== Cumulus Wind Speed Terminology === |
|||
How wind speeds relate to various weather stations is explained in FAQ. Basically there are 3 values processed by Cumulus: |
|||
#Gust - the highest wind speed over a particular period (this period defaults to 10 minutes), |
|||
#Average - the moving average of wind speed measurements over a particular period (this period can be set to what suits you, but defaults to 10 minutes), |
|||
#Latest - the most recent (wind speed or gust speed) measurement. |
|||
[[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Davis_stations.3F|FAQ: Wind speeds in Cumulus with Davis stations]], the ''EasyWeather'' differences in [[FAQ#What_do_the_various_wind_speeds_in_Cumulus_correspond_to.2C_with_Fine_Offset_stations.3F|FAQ: Wind speeds in Cumulus with Fine Offset stations]] and [[FAQ#How_do_I_set_up_recording_wind_speeds_in_Cumulus.2C_with_Oregon_Scientific_and_La_Crosse_stations| FAQ: Wind speeds with Oregon Scientific and La Crosse stations]] |
|||
When Cumulus is running normally, it actually stores (in the fifth item ''after'' the date in each entry, column F in a spreadsheet) the highest (moving) average in the period since the previous logger record. If Cumulus is not running, it stores the average from the station logger, that is the latest reported moving average. For this reason the most accurate wind speed records are achieved if Cumulus is running all day every day. |
|||
Cumulus calculates the monthly/annual/period ''average wind speed'' (as used for 'This ...' screens in '''View''' menu and the NOAA report) by trawling through the monthly logs, summing the values for those individually logged average wind speed, and dividing by the number of entries for the required calendar days. It therefore assumes that each entry has equal 'weight', i.e. each one covers the same period. If your monthly log(s) is/are a mixture of values stored while Cumulus is running and values copied from the station logger during periods when Cumulus was not running, then it is possible the two logging intervals have not been set to the same time, so the output value could be skewed for that month/year/period because of the difference in what was stored and the difference in the logging interval. |
|||
[[Windrun]] is also calculated from average wind speed measurements, in this case every minute if Cumulus is running, or from the values that are logged (at station logging interval) as average wind speed when Cumulus is restarted and catches up from the station logger. |
|||
===Using Monthly logs to deal with shorter (or incomplete) dayfile.txt records for particular dates=== |
|||
'''WORKAROUND if dates are present in monthly log and dayfile.txt but not all fields for that date exist in dayfile.txt''' |
|||
Example: '''add extra fields to records created by an earlier version of Cumulus''' [to help you, the versions (not builds) at which fields were added are indicated below]. |
|||
Second example: records imported prior to Cumulus processing them, so some calculated parameters (apparent temperature, heat index, rain rate, wind run) may not have been available from your weather station to insert in the log files. |
|||
''One method '''for Cumulus 1 only''' is (not near rollover time) to (1) take a copy of dayfile.txt original as backup, (2) in original file delete any days with partial information (e.g. from Cumulus versions that created fewer fields), (3) use ''create missing'' in the [[dayfile.txt]] selection in the '''Edit''' menu option of Cumulus (note whilst datafile.txt normally calculates all parameters like minimums and maximums from very frequent samples of your weather station, the resolution of create missing is limited to the interval between logging records in the monthly log), (4) rename the amended dayfile.txt as dayfile(generated).txt, (5) create a new dayfile.txt and (6) use a text editor to merge the required fields from the new dayfile(generated).txt with all other fields from (with reading access only) the backup copy of the original file. (This method preserves the original as a backup so you can experiment with different merges and do some cross-checking).'' |
|||
'''Cumulus MX''' If you use the facility mentioned earlier in this article to have MX updating the monthly logs into a database table, you can fairly easily update columns in this monthly table, and from it update columns in a daily summary table. See [https://cumulus.hosiene.co.uk/viewtopic.php?f=18&t=18096 this post in support forum] where I describe two PHP scripts designed for this type of task. |
|||
==List of fields in the file== |
|||
The table is split by Cumulus version for all changes since version 1.8.5; it shows: |
|||
*The field number (starting from zero to be consistent with index used for arrays in programming languages like JavaScript) and the equivalent letter that would be seen in a spreadsheet like Libre Office, Open Office, Excel etc are listed in first column. |
|||
*The second column shows an example, and the values shown are sometimes integers and sometimes floating point numbers, and this depends on a number of factors, including your station type. If you are editing the values manually, you should use integers for humidity, wind bearings, and the two solar radiation figures, and floating point for the others (add a ".0" if necessary). Note that the figure varies depending on the units you select, and your weather station may not have all the sensors needed, in that case for you some figures will always be zero. |
|||
*The third column describes the observation often with a link to where there is more information. |
|||
===Version 1.8.5:=== |
|||
{| border="1" class="wikitable" |
|||
|- |
|||
! style="width:60px" | Field # |
|||
! style="width:80px" | Example |
|||
! style="width:450px" | Description |
|||
|- |
|||
| 00(A) |
|||
|22/04/11 |
|||
| Date as ''2 figure day [separator] 2 figure month [separator] 2 figure year'' - the separator is that set in the windows system short date format (see [[setup]]) |
|||
|- |
|||
| 01(B) |
|||
|10:25 |
|||
| Current time |
|||
|- |
|||
| 02(C) |
|||
|8.1 |
|||
| Current temperature |
|||
|- |
|||
| 03(D) |
|||
|96 |
|||
| Current [http://en.wikipedia.org/wiki/Relative_humidity relative humidity] |
|||
|- |
|||
| 04(E) |
|||
|7.5 |
|||
| Current [http://en.wikipedia.org/wiki/Dewpoint dewpoint] |
|||
|- |
|||
| 05(F) |
|||
|13.5 |
|||
| Cumulus 'Average' wind speed (See [[#Cumulus_Wind_Speed_Terminology]]) |
|||
|- |
|||
| 06(G) |
|||
|20.3 |
|||
| Cumulus 'Gust' wind speed |
|||
|- |
|||
| 07(H) |
|||
|138 |
|||
| Average wind bearing (integer degrees) |
|||
|- |
|||
| 08(I) |
|||
|7.2 |
|||
| [[FAQ#How_is_my_rain_rate_calculated.3F | Current rainfall rate]] |
|||
|- |
|||
| 09(J) |
|||
|5.4 |
|||
| Total rainfall today so far (i.e. resets to zero at daily rollover) |
|||
|- |
|||
| 10(K) |
|||
|30.10 |
|||
| Current sea level pressure |
|||
|- |
|||
| 11(L) |
|||
|215.2 |
|||
| Total rainfall counter (content depends on weather station type, might be [[FAQ#It.27s_January_1st_and_my_.27rain_since_midnight.27_and.2For_.27last_24_hours_rain.27_are_wrong.2Fnegative | rainfall total so far this year]]. This figure is not intended to be useful in itself, it may vary for no apparent reason. It is used internally by Cumulus.) |
|||
|- |
|||
| 12(M) |
|||
|20.3 |
|||
| Inside temperature |
|||
|- |
|||
| 13(N) |
|||
|53 |
|||
| Inside humidity |
|||
|- |
|||
| 14(O) |
|||
|17.6 |
|||
| Cumulus 'Latest' gust (See [[#Cumulus_Wind_Speed_Terminology]]) |
|||
|- |
|||
| 15(P) |
|||
|4.8 |
|||
| [[wind chill]] (content depends on weather station type and Cumulus station settings) |
|||
|- |
|||
| 16(Q) |
|||
|8.1 |
|||
| [[Heat index]] |
|||
|- |
|||
| 17(R) |
|||
|0.5 |
|||
| UV Index (only valid if output by weather station) |
|||
|- |
|||
| 18(S) |
|||
|197 |
|||
| Solar Radiation (only valid if solar sensor on weather station) |
|||
|} |
|||
===Added in 1.9.1=== |
|||
{| border="1" class="wikitable" |
|||
|- |
|||
! style="width:60px" | Field # |
|||
! style="width:80px" | Example |
|||
! style="width:450px" | Description |
|||
|- |
|||
| 19(T) |
|||
|0.08 |
|||
| Evapotranspiration (only valid if output by weather station) |
|||
|- |
|||
| 20(U) |
|||
|171.88 |
|||
| Annual Evapotranspiration (only valid if ET sensor on weather station) |
|||
|- |
|||
| 21(V) |
|||
|3.4 |
|||
| [[Apparent temperature]] |
|||
|- |
|||
| 22(W) |
|||
|663 |
|||
| Current theoretical max solar radiation (see [[FAQ#What_is_this_.22Solar_Max.22_value_which_Cumulus_is_displaying.3F | the FAQ]]) |
|||
|- |
|||
| 23(X) |
|||
|3.1 |
|||
| Hours of sunshine so far today (only valid if solar sensor on weather station) |
|||
|} |
|||
===Added in 1.9.2=== |
|||
{| border="1" class="wikitable" |
|||
|- |
|||
! style="width:60px" | Field # |
|||
! style="width:80px" | Example |
|||
! style="width:450px" | Description |
|||
|- |
|||
| 24(Y) |
|||
|158 |
|||
| Current Wind bearing (See 7) |
|||
|} |
|||
===Added in 1.9.3=== |
|||
{| border="1" class="wikitable" |
|||
|- |
|||
! style="width:60px" | Field # |
|||
! style="width:80px" | Example |
|||
! style="width:450px" | Description |
|||
|- |
|||
| 25(Z) |
|||
|6.0 |
|||
| RG-11 rain today (only valid if output by sensor) |
|||
|} |
|||
=== Added in 1.9.4 === |
|||
{| border="1" class="wikitable" |
|||
|- |
|||
! style="width:60px" | Field # |
|||
! style="width:80px" | Example |
|||
! style="width:450px" | Description |
|||
|- |
|||
| 26(AA) |
|||
|5.7 |
|||
| Total Rainfall since midnight (this is for 0900/1000 'rollover' users; normally same as rain today for 'midnight rollover' users) |
|||
|} |
|||
=== Added in 3.6.0 === |
|||
{| border="1" class="wikitable" |
|||
|- |
|||
! style="width:60px" | Field # |
|||
! style="width:80px" | Example |
|||
! style="width:450px" | Description |
|||
|- |
|||
| 27(AB) |
|||
|13.7 |
|||
| [[Feels Like]] temperature |
|||
|} |
|||
==Example lines from the file== |
|||
Note that your field delimiters may be different, and your date delimiter too. |
|||
An extract of a few lines of the file (v.1.9.0) |
|||
Here logging is quarter hourly, the field delimiter is a semicolon, the decimal separator is a comma, and the date separator a dash. |
|||
30-09-10;19:00;16,4;94;15,4;5,2;13,3;17;3,6;21,0;995,3;47,7;25,6;62;6,1;16,4;16,4;0,0;0 |
|||
30-09-10;19:15;16,4;94;15,4;5,6;11,2;12;18,0;24,0;995,0;50,7;25,6;62;7,2;16,4;16,4;0,0;0 |
|||
30-09-10;19:30;16,2;94;15,2;7,9;15,8;355;7,2;25,8;994,3;52,5;25,7;62;12,2;16,2;16,2;0,0;0 |
|||
30-09-10;19:45;16,0;94;15,0;9,9;19,4;7;7,2;27,9;993,3;54,6;25,7;62;14,8;15,8;16,0;0,0;0 |
|||
30-09-10;20:00;15,9;94;14,9;12,4;20,9;354;7,2;30,0;993,0;56,7;25,7;62;19,4;15,4;15,9;0,0;0 |
|||
30-09-10;20:15;15,8;94;14,8;8,4;15,8;349;14,4;32,7;993,4;59,4;25,8;61;12,2;15,8;15,8;0,0;0 |
|||
30-09-10;20:30;15,4;94;14,4;13,8;33,1;317;28,8;40,5;993,7;67,2;25,8;61;23,4;14,7;15,4;0,0;0 |
|||
30-09-10;20:45;15,1;94;14,1;20,3;34,2;356;7,2;43,8;992,3;70,5;25,8;60;29,5;13,8;15,1;0,0;0 |
|||
30-09-10;21:00;15,3;94;14,3;20,2;35,6;358;10,8;46,8;991,0;73,5;25,8;60;28,1;14,0;15,3;0,0;0 |
|||
30-09-10;21:15;15,3;95;14,5;16,6;31,7;358;10,8;49,5;991,4;76,2;25,8;60;20,9;14,3;15,3;0,0;0 |
|||
30-09-10;21:30;15,3;94;14,3;14,0;27,0;324;18,0;54,3;992,3;81,0;25,8;60;15,8;14,5;15,3;0,0;0 |
|||
30-09-10;21:45;15,2;94;14,2;13,0;25,6;323;10,8;57,9;992,3;84,6;25,8;59;24,5;14,5;15,2;0,0;0 |
|||
30-09-10;22:00;15,0;94;14,0;16,7;31,7;312;10,8;60,6;993,0;87,3;25,8;59;23,4;13,9;15,0;0,0;0 |
|||
30-09-10;22:15;14,9;94;13,9;16,0;30,6;357;10,8;63,0;991,6;89,7;25,8;59;20,9;13,9;14,9;0,0;0 |
|||
30-09-10;22:30;14,9;94;13,9;17,6;31,7;3;3,6;63,3;990,6;90,0;25,8;59;19,4;13,7;14,9;0,0;0 |
|||
this is an example of a file from 1.9.1, with solar data: |
|||
Here logging is increased to every five minutes, the field delimiter is a comma, the decimal separator is a full stop, and the date separator is also a full stop. |
|||
22.04.11,10:25,8.1,96,7.5,13,20,138,0.0,0.0,1013.24,215.2,20.3,53,17,4.8,8.1,0.0,197,0.08,171.88,3.4,663,0.0 |
|||
22.04.11,10:30,8.1,96,7.5,13,20,142,0.0,0.0,1013.28,215.2,20.2,53,11,4.8,8.1,0.0,216,0.08,171.88,3.3,673,0.0 |
|||
22.04.11,10:35,8.1,96,7.5,12,18,142,0.0,0.0,1013.31,215.2,20.2,53,11,5.1,8.1,0.0,227,0.08,171.88,3.8,682,0.0 |
|||
and from version 1.9.2: |
|||
Here logging is at the default interval, and the delimiters have their default settings for UK systems. |
|||
01/10/11,04:40,18.0,75,13.5,5.0,6.7,169,0.0,0.0,1021.4,885.3,24.4,61,5.6,18.0,18.0,0.0,0,0.00,0.00,17.5,0,0.0,158 |
|||
01/10/11,04:50,17.9,75,13.4,4.5,9.5,169,0.0,0.0,1021.4,885.3,24.4,61,2.0,17.9,17.9,0.0,0,0.00,0.00,17.5,0,0.0,180 |
|||
01/10/11,05:00,17.8,75,13.3,5.1,9.5,177,0.0,0.0,1021.2,885.3,24.4,61,7.5,17.8,17.8,0.0,0,0.00,0.00,17.2,0,0.0,180 |
|||
[[Category:Log Files]] |
|||
Latest revision as of 23:17, 31 December 2021
Please see Standard log files, Extra Sensor Files, Air_Link_Log.txt, Month.ini, or Monthlyalltime.ini, as relevant.
Please be aware that all of the above links lead to documentation about files that may be described as monthly log files when reading individual Cumulus forum and wiki references.
Potential Name Confusion
Those who used early versions of Cumulus will remember that the term Log File was originally used for Log.xml, but Steve Loft later accepted that all the files with a .ini file extension could be described as log files, so he introduced the term Weather Diary. A bit later, the detailed data file, began to be also called log files, and the confusion grew. When Extra Sensor Files were added, these also became log files. The Wiki page now called "Standard Log Files" was first called "Log files", then became "Monthly Log Files" because "Log files" became a category. Then "Extra Sensor Files" were added and "standard log files" replaced "Monthly Log Files" to make it clearer which file held the standard values, and which file held values for extra sensors.
Steve Loft wrote the Cumulus software mainly for his own use, although after a while he let others use it and added features that others requested. Cumulus was developed in his spare time outside a full-time employment, and consistency and terminology were not issues he worried about.
During the development of the original Cumulus software (Cumulus 1 and 2) by Steve Loft, the name "monthly log" was most frequently used as a collective term for the last two (.ini) files linked above because those files tracked the monthly extreme records.
When Steve Loft was working on his MX Beta, he decided to implement within MX functionality the database schema that Mark Crossley had designed on this page, where Mark selected the name Monthly for the table. David A Jamieson created versions 1.0 and 1.1 of ImportCumulusFile for creating a single table called dayfile, but Mark extended it to cover realtime and what he called monthly log files. That means the standard log can in MX optionally feed data into a database table called Monthly by default in MX.
This database table terminology can be misleading as that database table is not organised by month in any way, a single table contains every logged line that has been inserted into it by Steve's ExportMySQL.exe, Mark's ExportToMySQL.exe, or by CumulusMX.exe itself. In fact MX allows you to give this table another name, which it will then use in all SQL.
Mark Crossley has taken over development of MX, the terminology "monthly log" is now being used for the first linked file (standard logs) instead of for month.ini the file that is truly logging monthly extremes. The name has been adopted simply because a new log file is started for each month.
For novices, there can be a feeling of confusion:
- The new practice leaves "month.ini" without a descriptive name
- Each "dayfile" line represents a new day so the filename matches the content, but a line in "Oct16log.txt" does not represent a month, and the file name does not contain the phrase "month"
- The names "Month.ini" and "Monthlyalltime.ini" do contain the phrase "month", they do contain monthly data in each line, and do log extreme records that clearly identify with a particular month
A future Wiki editor may decide on a different strategy, but at the moment the Cumulus Wiki has returned to the original Steve Loft terminology for each of the log files, for consistency with older articles in this Wiki, and the oldest posts on the forum.
Enhancement never implemented
Steve Loft was several times asked if he could change the design of his software and what it stored in Standard log files, Extra Sensor Files and the other detailed logging files.
All these files currently periodically store spot readings. Some of the competition in weather software, periodically store the highest and lowest since the previous line was stored.
One advantage of the alternative approach is that you never lose any extremes. Another advantage is the alternative allows people to report extremes for parts of days. For example, some people wanted 9 am to 9 pm reported separately to night time extremes, in other nations, the dividing time was 3 pm not 9 pm.
Steve Loft would respond he had made many design decisions that were not ideal, but the software was based on his own needs. If people wanted drastic changes, they should do what he had done, and write software that did what suited them.
Yet it appears he did plan to implement this suggestion in MX, see Cumulusmx.db#Releases 3.0.0 to 3.11.4 inclusive.
