Your Own Server: Difference between revisions

From Cumulus Wiki
Jump to navigationJump to search
mNo edit summary
(Updated wiki page for MX changes since 3.6.0 when page was created)
Line 31: Line 31:


Cumulus MX includes two SQLite databases;
Cumulus MX includes two SQLite databases;
#The first database was added by Steve Loft, but he never documented what it is used for (see [[Cumulusmx.db]]) and in the support forum when someone asked, nobody was able to answer.
#The first database was added by Steve Loft, but he never documented what the first table in that database was to be used for (see [[Cumulusmx.db]]). The second table in that database is used for [[Recent history]].
#*It still remains in MX, so it must be used for something. One could hazard a guess that as MX needs to maintain a lot of numbers (the [[Correcting Extremes|extremes]] it monitors, the numbers it holds for the [[Cumulus_MX_Local_API|admin interface]], the numbers it can output for [[Highcharts_-_Recent|recent graphs]] and [[Highcharts_-_Historic|historic graphs]] amongst others) this might be why it needs a database
#*If you do find out what the first database is used for, please update the Wiki article on it!
#The second database is the Weather Diary [[Weather Diary|documented here]] added by Mark Crossley. Also see [https://cumulus.hosiene.co.uk/viewtopic.php?f=37&t=17919 in the support forum] for a topic comparing the differences between Cumulus 1 and Cumulus MX weather diaries.
#The second database is the Weather Diary [[Weather Diary|documented here]] added by Mark Crossley. Also see [https://cumulus.hosiene.co.uk/viewtopic.php?f=37&t=17919 in the support forum] for a topic comparing the differences between Cumulus 1 and Cumulus MX weather diaries.
#* The [[MX Administrative Interface|Admin Interface]] provides a page to view, or edit, a single date in the weather diary. You can install another tool that edits SQLite databases if you want (see next section).
#* The [[MX Administrative Interface|Admin Interface]] provides a page to view, or edit, a single date in the weather diary. You can install another tool that edits SQLite databases if you want (see next section).
Line 39: Line 37:
The legacy Cumulus software used a [[Log.xml|XML file]] to store the weather diary.
The legacy Cumulus software used a [[Log.xml|XML file]] to store the weather diary.


===Reading content of SQLite databases===
===Two ways for editing the MX Weather Diary===


Obviously, Cumulus MX can read and update both of these databases, but there are a number of software packages available that can be used outside MX to access these database files, and to minimise technical detail, two will be named as possible candidates, you will find others online.
You can install phpLiteAdmin (the significant part of that name is in the middle) to read (you can display the whole table very easily) and edit (via a friendly GUI) either, or both, of the SQLite databases that MX uses as mentioned above.


On Microsoft Windows, install phpLiteAdmin (the significant part of that name is in the middle) to read (you can display the whole table very easily) and edit (via a friendly GUI) either, or both, of the SQLite databases that MX uses as mentioned above.
Install ‘’’phpLiteAdmin’’’ using <code>sudo apt install phpliteadmin</code>. Please see [[MX_on_Linux#The_various_components_to_commands_for_installation]] for an explanation of that instruction.


In Linux use <code>sudo apt install sqlitebrowser</code> to install easy-to-use software to read/edit these databases.
On a Linux computer, the main script for running ‘’’PhpLiteAdmin’’’ is located at '''/usr/share/phpliteadmin/phpliteadmin.php'''. Also in the '''/usr/share/phpliteadmin/’’’ folder, there is a link to ‘’’phpliteadmin.config.php’’’ which is actually in the ‘’’/var/lib/phpliteadmin/’’’ folder.


You may need to change file permissions.
By default ‘’phpLiteAdmin’’ expects any database it accesses to be in a single folder, and to have a file extension ‘’’.lite’’’. ‘’’phpliteadmin.config.php’’’ defines the folder where it expects to find the database as in ‘’’/var/lib/phpliteadmin/’’’ folder by default. ‘’’ phpliteadmin.php’’’ defines the folder where it expects to find the database as in '''/usr/share/phpliteadmin/’’’ folder by default.


Note that sqlite databases use sequential file structure, not a relational database design. This implies for updates, rows are physically identified by row number (order of creation), (not by the primary key that SQL accesses use with relational database designs).
This means you will need to follow the instructions in that main script to change the password, and to define the path to the database you want to read, as well as telling it the file name extension MX uses (both databases have the wrong extension to be found automatically).

You may need to change file permissions, and you may decide to use the symbolic link example given [[MX_on_Linux#folder_commands|here]]] so MX accesses the diary at a location that suits phpLiteAdmin.

I find the admin interface is not always reliable for updating the weather diary, MX uses a very simple way to access hardware;
*when a successful insert or update happens, a confirmation message is returned
*when it fails to store an update, there is no error either on the editing screen or in the MXDiags file.

The admin interface has a calendar type interface to pick the entry you want to insert or update; this is probably quickest for any edits in current month. The phpliteadmin graphical interface will list the whole database table and let you edit (or delete) any row, you can also insert (to create a new row), or import/export (using SQL), or copy the whole database.

Note that sqlite databases use sequential file structure and so for updates rows are physically identifed by row number (order of creation), not by the primary key that most SQL accesses use.

I can use phpliteadmin, or the admin interface editor, for making changes to my Weather Diary database now. Both can be accessed from a browser on any of my devices.


= Adding a web server and a database server for your own purposes =
= Adding a web server and a database server for your own purposes =
Line 78: Line 64:


To get maximum benefit when using with MX, you need multiple components:
To get maximum benefit when using with MX, you need multiple components:
# web server – there are various options listed below; beware "apache2" is huge as it is very complicated, but secure; other options consume fewer resources, and may be easier to use, but they will also be more restrictive
#web server – apache2 is just one ofvarious options listed below
#PHP script - from [https://windows.php.net/download/]
# PHP script - from [https://windows.php.net/download/] - this provides a way of controlling what SQL is run depending on other factors
#Database server - MySQL Community edition from [mysql.com] or MariaSQL from [https://downloads.mariadb.org/].
# Database server - MySQL Community edition from [mysql.com] or MariaSQL from [https://downloads.mariadb.org/].
#Integrating components like ‘’’phpMyAdmin’’ (there are alternatives) from [https://www.phpmyadmin.net/downloads/] which combines PHP and MySQL
# Integrating components like ‘’’phpMyAdmin’’ (there are alternatives) from [https://www.phpmyadmin.net/downloads/] which combines PHP and MySQL


There are a number of packages that combine all of these:
There are a number of packages that combine all of these:
Line 91: Line 77:
I will give a few examples, but there are many more, each has its supporters, but also each has people who criticise it. You can search on-line and find both all the (good and bad) comments, and clear instructions.
I will give a few examples, but there are many more, each has its supporters, but also each has people who criticise it. You can search on-line and find both all the (good and bad) comments, and clear instructions.


This page will only describe one installation (on a Raspberry Pi) in any detail, otherwise it will become far too complex.
This Wiki page will only describe one installation (on a Raspberry Pi) in any detail, otherwise it will become far too complex.


=== Install Apache 2 (or another web server) ===
=== Install a web server ===


I will first mention 3 possible web servers than can run on a Microsoft Windows personal computer (or server box):
I will first mention 3 possible web servers than can run on a Microsoft Windows personal computer (or server box):
Line 100: Line 86:
#You might choose to install the Abyss web server from [aprelium.com]
#You might choose to install the Abyss web server from [aprelium.com]


If you get components separately, then there is extra work to get them to work together, but you will find clear explantions on download sites. Generally, if you have an all-in-one package, it seems easier; and normally upgrading one component is also easy (providing its a minor version upgrade, it might be more complex with a major version upgrade).
If you get components separately, then there is extra work to get them to work together, but you will find clear explanations on download sites. Generally, if you have an all-in-one package, it seems easier; and normally upgrading one component is also easy (providing its a minor version upgrade, it might be more complex with a major version upgrade).


I will next mention 3 possible web servers than can run on a Linux computer:
I will next mention 3 possible web servers than can run on a Linux computer:
#You might choose '''nginx''', as it is a quite popular web server for small computers (like the RPi), install this web server with <code>sudo apt-get install nginx -y</code>.
#You might choose '''Apache 2''' as it is probably the most comprehensive, so if you have enough space, install it using <code>sudo apt install apache2 -y</code>.
#You might choose '''Apache 2''' as it is probably the most comprehensive, so if you have enough space, install it using <code>sudo apt install apache2 -y</code>.
#*You might want to add Fast CGI to your apache server (this is a way of interfacing interactive programmes with a web server, but if you don’t have any idea what that might mean, skip this)
#*You might want to add Fast CGI to your apache server (this is a way of interfacing interactive programmes with a web server, but if you don’t have any idea what that might mean, skip this)
#* To add this use <code>sudo apt install libapache2-mod-fcgid</code>.
#* To add this use <code>sudo apt install libapache2-mod-fcgid</code>.
#You might choose '''nginx''', as it is a quite popular web server for small computers (like the RPi), install this web server with <code>sudo apt-get install nginx -y</code>.
#You might choose '''lighttpd''', as a web server that is designed to use as little space on your Linux computer as possible, install it with <code>sudo apt-get install lighttpd -y</code>.
#You might choose '''lighttpd''', as a web server that is designed to use as little space on your Linux computer as possible, install it with <code>sudo apt-get install lighttpd -y</code>.


Line 115: Line 101:
As already mentioned, some web servers come in a package that includes this script language. Again, as mentioned before, such packages may be rarely upgraded so the included PHP might be an old version with bugs or security concerns.
As already mentioned, some web servers come in a package that includes this script language. Again, as mentioned before, such packages may be rarely upgraded so the included PHP might be an old version with bugs or security concerns.


At time of updating this Wiki page, PHP 8.1 is stable version, PHP 8.2 is available to test, and PHP 9.0 is planned. Older versions 7.x, and 5.x are largely obsolete, although some web providers may still make them available.
Whether you are installing PHP because you don’t have it, or because you want to upgrade it:

* On Microsoft windows down load a zip from from [https://windows.php.net/download/]. You will want a “multi-threaded” version to use with a web server, as several PHP web pages may need to run at the same time.
* PHP is not the only script language available, but it is quite comprehensive
** PHP can be used either in a fairly simple way by non-technical people
** or in an object-oriented way for those more technical to achieve success with more complex scripts.
* To check which PHP modules have been installed by any command, type <code>php -m</code>.
* To test that php is installed, type <code>php -v</code> into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.
* If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least '''mysqli''' and '''mbstring'''.




====Microsoft Windows - download/install PHP====

Straight forward, just down load a zip from from [https://windows.php.net/download/]. You will want a “multi-threaded” version to use with a web server, as several PHP web pages may need to run at the same time.

*PHP is not the only script language available, but it is quite comprehensive
**being able to be used either in a fairly simple way by non-technical people
**or in an object-oriented way for those more technical to achieve success with more complex scripts.
*To check which PHP modules have been installed by the above command, type <code>php -m</code>.
*If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least '''mysqli''' and '''mbstring'''.


To test that php is installed, type <code>php -v</code> into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.
====Linux operating systems - best repository====

The default repositories (see [[Preparing_your_Linux_computer_for_MX#The_various_components_to_commands_for_installation|here]]) may not contain up-to-date versions of all software, or may contain some software that is not compatible with other software.

You can add another repository that generally has reasonably new PHP versions (a few weeks behind the official [https://www.php.net/downloads.php] site) using
<pre>sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt update</pre>


* On Linux, the simplest instruction to install it is <code>sudo apt install php -y</code>, although in practice this will not install all the PHP modules you actually need
* On Linux, the simplest instruction to install it is <code>sudo apt install php -y</code>, although in practice this will not install all the PHP modules you actually need
** A warning, which version of PHP that command gives you depends on which php version is found first in the [[MX_on_Linux#The_various_components_to_commands_for_installation|respositories]] your computer has in its source list, so you may end up with the oldest!
** A warning, which version of PHP that command gives you depends on which php version is found first in the [[MX_on_Linux#The_various_components_to_commands_for_installation|respositories]] your computer has in its source list, so you may end up with the oldest!
* A better variarion is to actually list the version we want:
* A better variation is to actually list the version we want:
<code>sudo apt install php''V.x''</code> (where V is major version, and x is minor version e.g. 7.4 or 8.1)
** This may still not give us the modules we want, but it does control which version is obtained
* Therefore the best variation is to list both the modules we want, and the version we want (here "x" denotes minor version, adjust to what is latest stable (e.g. change to php8.1-common etc.):
<code>sudo apt install -y php8.x-common php8.x-cli php8.x-json php8.x-xml php8.x-readline php8.x-mbstring php8.xmysql php8.x-curl php8.x-gd pdp8.x-psqlite php8.x-pdo -y</code>
* The Linux package manager will ensure that any components, that are required by dependencies, but not listed specifically, are also installed
** Loading from the "packages.sury.org/php" repository has one disadvantage, the dependencies there don't automatically integrate the PHP with your web server (if you are using apache 2 for web server, add "libapache2-mod-php8.x" to list);

====Linux operating systems - standard repository====

* On Linux, the simplest instruction to install PHP is <code>sudo apt install php -y</code>, although in practice this will not install all the PHP modules you actually need
** A warning, which version of PHP that command gives you depends on which php version is found first in the [[MX_on_Linux#The_various_components_to_commands_for_installation|respositories]] your computer has in its source list, so you may end up with the oldest!
* A better variation is to actually list the version we want:
<code>sudo apt install php7.4</code>
<code>sudo apt install php7.4</code>
** This still does not give us the modules we want, but it does control which version is obtained
** This still does not give us the modules we want, but it does control which version is obtained
*A still better variation is to list the components we want:
*A still better variation is to list the components we want:
<code>sudo apt install libapache2-mod-php7.4 php7.4 php7.4-cli php7.4-common php7.4-json php7.4-xml php7.4-opcache php7.4-readline php7.4-mbstring php7.4mysql php7.4-curl php7.4-gd pdp7.4-psqlite php7.4-pdo -y</code>
<code>sudo apt install php7.4 php7.4-cli php7.4-common php7.4-json php7.4-xml php7.4-opcache php7.4-readline php7.4-mbstring php7.4mysql php7.4-curl php7.4-gd php7.4-sqlite php7.4-pdo -y</code>
** ”libapache2-mod-php7.4” is just part of what integrates apache 2 and PHP
** Add ”libapache2-mod-php7.4” as just part of what integrates apache 2 and PHP
** “php7.4-cli” is what allows us to run scripts in batch
** The included “php7.4-cli” is what allows us to run scripts in batch
** “php7.4-common” and “php7.4-mbstring” are pre-requisites for various other modules, and the second is used by ‘’’phpMyAdmin’’’ and other tools
** “php7.4-common” and “php7.4-mbstring” are pre-requisites for various other modules, and the second is used by ‘’’phpMyAdmin’’’ and other tools
** “php7.4-json” (see [[Php_webtags#Option_3:_JavaScript_Object_Notation|here]] and [[:Category:JSON_Files#Data_Transfer_Format]] and “php7.4-xml” (see [[Php_webtags#Option_1:_eXtensible_Markup_Language_alternative|this section]]) are used to handle particular file types
** “php7.4-json” (see [[Php_webtags#Option_3:_JavaScript_Object_Notation|here]] and [[:Category:JSON_Files#Data_Transfer_Format]] and “php7.4-xml” (see [[Php_webtags#Option_1:_eXtensible_Markup_Language_alternative|this section]]) are used to handle particular file types
** “php7.4mysql” integrates php and MariaSQL/MySQL
** “php7.4mysql” integrates php and MariaSQL/MySQL
** "php7.4-sqlite php7.4-pdo" are needed to access the weather diary or recent history databases
** “php7.4-curl” “php7.4-gd” are used by some third-party routines designed by Cumulus contributors
** “php7.4-curl” “php7.4-gd” may be needed as these are used by some third-party routines designed by Cumulus contributors
*You can add another repository that generally has fairly new versions (a few weeks behind the official [https://www.php.net/downloads.php] site) using
* There are advantages in using default repositories:
<code>sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt update</code>
**With this repository, we can install the equivalents in PHP8.0, here they are:
<code>sudo apt install -y php8.0-common php8.0-cli libapache2-mod-php8.0 php8.0-json php8.0-xml php8.0-readline php8.0-mbstring php8.0mysql php8.0-curl php8.0-gd pdp8.0-psqlite php8.0-pdo -y</code>
* The Linux package manager will ensure that any components, that are required by dependencies, but not listed specifically, are also installed
** Loading from the "packages.sury.org/php" repository has one disadvantage, the dependencies there don't integrate the PHP with your web server;
** installing a web server and PHP from the default repositories will pick up joint dependencies and thus integrate the web server and php.
** installing a web server and PHP from the default repositories will pick up joint dependencies and thus integrate the web server and php.
*PHP is not the only script language available, but it is quite comprehensive
**being able to be used either in a fairly simple way by non-technical people
**or in an object-oriented way for those more technical to achieve success with more complex scripts.
*To check which PHP modules have been installed by the above command, type <code>php -m</code>.
*If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least '''mysqli''' and '''mbstring'''.


To test that php is installed, type <code>php -v</code> into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.


===Creating a Home web page on your web server===
===Creating a Home web page on your web server===
Line 180: Line 196:
To install this database server, we type <code>sudo apt install mariadb-server php-mysql -y</code>.
To install this database server, we type <code>sudo apt install mariadb-server php-mysql -y</code>.


Our database files will be stored at (Linux) '''/var/lib/mysql''' by default. Our MariaDB configuration is stored at '''/etc/mysql/mariadb.conf.d/50-server.cnf''', and it is the ''datadir='' entry that controls where the database files are stored.
Our database files will be stored at (Linux) '''/var/lib/mysql''' by default. Our MariaDB configuration is stored at '''/etc/mysql/mariadb.conf.d/50-server.cnf''', and it is the ''datadir='' entry that controls where the database files are stored, so edit that to store files somewhere else, but make sure the storage location has right ownership permissions. This Wiki page will not give you the full technical details, so only those with some technical knowledge do such changes.


=== Making your database secure ===
=== Making your database secure ===
Line 198: Line 214:
===Getting web and database servers ready for use===
===Getting web and database servers ready for use===


We need to create a user for PhpMyAdmin (or adMiner or whatever) to access our database and another for Cumulus to use to access the database tables. At the moment our database access has the single root@localhost user we created when we installed MariaDB. The initial password was set then, and we need to use it to get access to MariDB monitor where we can insert some SQL commands to create these two users.
We need to create a user for PhpMyAdmin (or adMiner or whatever) to access our database and another for Cumulus to use to access the database tables. At the moment our database access has the single '''root@localhost''' user we created when we installed MariaDB. The initial password was set then, and we need to use it to get access to MariDB monitor where we can insert some SQL commands to create these two users.


PhpMyAdmin on first start up will ask for username (here I choose "admin") and password, thereafter it will use same log-in (by default you see log-in screen each time you restart or if it is left idle for a long time), let us create a user called 'admin' for it.
Most tools, like PhpMyAdmin, on first start up will ask for username (here I choose "admin") and password, thereafter the tool should use same log-in (by default you see log-in screen each time you restart or if it is left idle for a long time), let us create a user called 'admin' for it.


The database name (here I choose "cumulus" for the data base name), user name (here I choose "weather" for the user name), and password, must all match those set in MX using the [[MX_Administrative_Interface#MySQL_settings|MySQL_settings]] in the admin interface. You could of course use PhpMyAdmin (or AdMiner) to create additional user names, and to create the database, but I assume all is done in the following bit of SQL. Remember, the Windows operating system is not case sensitive, but all Linux based operating systems are case sensitive (so whatever pattern of capitals and lowercase you choose must be used every time. Also all names must start with a letter, can contain only letters or digits (no punctuation), and must not be a reserved word ("password" is a reserved word, so you cannot use that for a password, nor for a column name).
The database name (here I choose "cumulus" for the data base name), user name (here I choose "weather" for the user name), and password, must all match those set in MX using the [[MX_Administrative_Interface#MySQL_settings|MySQL_settings]] in the admin interface. You could of course use PhpMyAdmin (or AdMiner) to create additional user names, and to create the database, but I assume all is done in the following bit of SQL. Remember, the Windows operating system is not case sensitive, but all Linux based operating systems are case sensitive (so whatever pattern of capitals and lowercase you choose must be used every time. Also all names must start with a letter, can contain only letters or digits (no punctuation), and must not be a reserved word ("password" is a reserved word, so you cannot use that for a password, nor for a column name).
Line 224: Line 240:
ExportMySQL.exe and CumulusMX.exe (see [[Cumulus_MX#Executables]] for details) both create SQL for updating tables in a MySQL database, such as the one our MariaDB software package we have installed can create.
ExportMySQL.exe and CumulusMX.exe (see [[Cumulus_MX#Executables]] for details) both create SQL for updating tables in a MySQL database, such as the one our MariaDB software package we have installed can create.


This option is about installing a package to do the tasks that go with operating a database, that MX may not be able to do in the way you require.
However, there is nothing in the MX distribution that lets us back up and manipulate (e.g. delete rows with errors, or correct rogue numbers in a column) tables in this database. This option is about installing a package to do the tasks that go with operating a database.


====PhpMyAdmin====
====PhpMyAdmin====
Line 242: Line 258:
#The version of phpmyadmin in the default repository is not compatible with PHP7.2 and above
#The version of phpmyadmin in the default repository is not compatible with PHP7.2 and above
# Visit the phpMyAdmin [https://www.phpmyadmin.net/downloads/ official download page] and look for the .tar.gz URL and download it using wget.
# Visit the phpMyAdmin [https://www.phpmyadmin.net/downloads/ official download page] and look for the .tar.gz URL and download it using wget.
# The following steps assume version 5.1.0, released Feb 2021 is to be downloaded. (If a later version is now available, make sure to change the commands below to match)
# The following steps assume version 5.2.0, released in 2022 is to be downloaded. (If a later version is now available, make sure to change the commands below to match)
# Decide whether you just want English, or need another language
# Decide whether you just want English, or need another language
#* For English only, the wget command is <code>sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.1.0/phpMyAdmin-5.1.0-english.tar.gz</code>
#* For English only, the wget command is <code>sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.2.0/phpMyAdmin-5.2.0-english.tar.gz</code>
#* For other languages, the wget command is <code>sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.1.0/phpMyAdmin-5.1.0-all-languages.tar.gz</code>
#* For other languages, the wget command is <code>sudo wget https://files.phpmyadmin.net/phpMyAdmin/5.2.0/phpMyAdmin-5.2.0-all-languages.tar.gz</code>
# Change to folder where we will extract our download <code>cd /usr/share/</code>
# Change to folder where we will extract our download, this could be <code>cd /usr/share/</code> or some other directory perhaps on an extra storage drive
#* For English only, extract your download into the current folder using <code>sudo tar xzf phpMyAdmin-5.1.0-english.tar.gz</code>
#* For English only, extract your download into the current folder using <code>sudo tar xzf phpMyAdmin-5.2.0-english.tar.gz</code>
#* For other languages, extract your download into a sub-folder of the current folder using <code>sudo tar xzf phpMyAdmin-5.1.0-all-languages.tar.gz</code>
#* For other languages, extract your download into a sub-folder of the current folder using <code>sudo tar xzf phpMyAdmin-5.2.0-all-languages.tar.gz</code>
# Create a folder to hold the download <code>sudo mkdir /usr/share/phpmyadmin</code>
# Create a folder to hold the download within the folder where we did our download <code>sudo mkdir phpmyadmin_5.2.0</code>
# Now we will move the contents from the sub-folder created by the "tar" command into the folder we just created
# Now we will move the contents from the sub-folder created by the "tar" command into the folder we just created
#* For English only, the move command is <code>sudo mv phpMyAdmin-5.1.0-english/* /usr/share/phpmyadmin</code>
#* For English only, the move command is <code>sudo mv phpMyAdmin-5.2.0-english/* ./phpmyadmin_5.2.0</code>
#* For other languages, the move command is <code>sudo mv phpMyAdmin-5.1.0-all-languages/* /usr/share/phpmyadmin</code>
#* For other languages, the move command is <code>sudo mv phpMyAdmin-5.2.0-all-languages/* ./phpmyadmin_5.2.0</code>
# There is some configuration to do, for simplicity here assume the destination directory ("./phpmyadmin_5.2.0") is actually "/usr/share/phpmyadmin" and that we will use '/var/lib/' and '/etc/' to store other directories used by phpmyadmin:
# There is some configuration to do
#* First <code>sudo nano /usr/share/phpmyadmin/libraries/vendor_config.php</code> to edit (with nano) the vendor configuration
#* First <code>sudo nano /usr/share/phpmyadmin/libraries/vendor_config.php</code> to edit (with nano) the vendor configuration
#** change the path shown in the existing line mentioning 'TEMP_DIR' so it becomes <code>define('TEMP_DIR', '/var/lib/phpmyadmin/tmp/');</code>
#** change the path shown in the existing line mentioning 'TEMP_DIR' so it becomes <code>define('TEMP_DIR', '/var/lib/phpmyadmin_5.2.0/tmp/');</code>
#** change the path shown in the existing line mentioning 'CONFIG_DIR' so it becomes <code>define('CONFIG_DIR', '/etc/phpmyadmin/');</code>
#** change the path shown in the existing line mentioning 'CONFIG_DIR' so it becomes <code>define('CONFIG_DIR', '/etc/phpmyadmin/');</code>
#The following sequence of commands will
#The following sequence of commands will
#* add the mysqli module to our php install, <code>sudo phpenmod mysqli</code>
#* add the mysqli module to our php install, <code>sudo phpenmod mysqli</code>
#*will restart apache, <code>sudo service apache2 restart</code>
#*will restart apache (change if you are using a different web server), <code>sudo service apache2 restart</code>
#*will create a symbolic link for the phpadmin installation in the server web root so it can be seen (and used) in our browser, <code>sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin</code>
#*will create a symbolic link for the phpadmin installation in the server web root so it can be seen (and used) in our browser, <code>sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin</code>, note the first path is to directory where we stored our phpmyadmin files, and the second path/file is the default web directory and the symbolic link file name, adjust these paths to what is needed for you.
#*will give the standard user (pi) ownership of the database files and the web pages: <code>sudo chown -R pi:www-data /var/www/html/</code>
#*will give the standard user (pi) ownership of the database files and the web pages: <code>sudo chown -R pi:www-data /var/www/html/</code>
# phpMyAdmin now has the files where it expects to see them, and should work
# phpMyAdmin now has the files where it expects to see them, and should work
Line 283: Line 299:
===Populating your database tables on your Pi===
===Populating your database tables on your Pi===


Cumulus MX has functionality to update database tables at one of 3 intervals:
Cumulus MX has functionality to update database tables at one of several intervals:
#real-time
# fixed SQL at real-time
#standard interval
# fixed SQL at standard interval
#end of day
# fixed or custom SQL at end of day
# custom number of seconds
# custom number of minutes


MX now has ability to run SQL updates when you edit a log file, provided you use the predefined column and table names. MX also has the ability to store failed SQL and to try to run it again at the next rollover. This will fail again and again until MX is stopped, if the SQL is specified wrongly, but is designed to cope with correct SQL, that fails because the database server is down, and assumes the database server may be working again by next rollover.
The database tables can use the column names in the schemas pre-defined by Cumulus MX or in a custom schema (where you specify the column names). The settings are all in [[MX_Administrative_Interface#MySQL_settings]], so read that section to find out more.


The database tables can use the column names in the schema pre-defined by Cumulus MX for the predefined queries, or in a custom schema (where you specify the column names) if you specify the required SQL. The settings are all in [[Cumulus.ini#Optional_Structured_Query_Language_Settings|SQL settings]], so read that cross-reference to find out more.
You might have started using MX before you set up your database. There is a option in that part of the admin interface to create database tables (as required) for each of the 3 updating intervals. For example, the default name for the table updated at the standard interval is "monthly", but you could give it a name of "standard" or whatever you like.


You might have started using MX before you set up your database. There is a option in that part of the admin interface to create database tables (as required) for each of the first 3 updating intervals. For example, the default name for the table updated at the standard interval is "monthly", but you could give it a name of "standard" or whatever you like.
The MX release distribution includes another executable. Type <code>cd CumulusMX && sudo mono ExportMySQL.exe monthly</code> to run the executable. The first parameter is "monthly" even if the table has a different name.

#if the table name is defined in the admin interface, and the table already exists in the database with the correct columns defined, then the executable will use that table.
#There is an optional second parameter that specifies the log file name to read.
There is an optional utility that can be installed. Type <code>cd CumulusMX && sudo mono ExportToMySQL.exe parameter</code> to run the executable. The mandatory parameter specifies the log file name to read (dayfile.txt or standard monthly log file).
#*if the second parameter is not specified, this executable will look at every log file (in "data" folder and with file name that starts with month, then has "log.txt"),
# for each line in the log file the executable will try to insert a row in the database table
# for each line in the log file the executable will try to insert a row in the database table
#*the SQL syntax used is "insert ignore", so if the row for that log file line already exists in the table, it will skip onto next line of log file.
#*the SQL syntax used is "insert ignore", so if the row for that log file line already exists in the table, it will skip onto next line of log file.

Similar instructions apply for the end of day table, although as there is only one daily summary log file, there is no optional second parameter, just type <code>cd CumulusMX && sudo mono ExportMySQL.exe dayfile</code>.


There is no way to use this executable for insert of past rows into the real-time table.
There is no way to use this executable for insert of past rows into the real-time table.
Line 315: Line 330:
=== Restarting Web Server ===
=== Restarting Web Server ===


After all these installs, we need to restart Apache (so it loads the PHP and MariaDB), by typing <code>sudo service apache2 restart</code> or (if we just want the Apache configuration reloaded) by typing <code>sudo systemctl reload apache2</code>. Similar commands apply for other web servers.
After all these installs, we need to restart your web server (so it loads the PHP and MariaDB), if you have installed apache 2 do this by typing <code>sudo service apache2 restart</code> or (if we just want the Apache configuration reloaded) by typing <code>sudo systemctl reload apache2</code>. Similar commands apply for other web servers.


We will need to restart Apache (or whichever web server we installed) any time we change our php.ini files, database passwords, and anything else that is checked when the web server starts.
We will need to restart Apache (or whichever web server we installed) any time we change our php.ini files, database passwords, and anything else that is checked when the web server starts.

Revision as of 18:00, 16 June 2022

Crystal Clear info.png This document is 'Work In Progress' so content may not be complete.

Request for help from Wiki Readers

  • Do you understand how MX works?
  • Do you use hardware, or MX functionality, that is not yet documented? Can you begin that documenting?
  • Can you contribute simple text for novice users, examples of what you have done, correction of typing or factual errors, or supply missing details?
  • Will you make this page more useful by bringing content up-to-date as new releases change some information written for older releases?
  • Does any page need a section for novices, so they don't need to read more technical information further down that page?
  • Is there some information on this page, that should be on a separate page? Can you create the new page and move the less relevant information off this page, don't forget this page needs a link to the new page so people who expect to find it here know where it has moved to?

If you plan on contributing to the Wiki, then you will need an account.

  • Please use the Request Account form to apply for an account. Note that the Wiki is currently undergoing restructuring and is largely locked for editing, but please apply for an account if you wish to contribute in the future.
  • You will find help on how to contribute to this wiki at How to Edit.
  • If you need to consult others, please use the Cumulus Wiki suggestions forum.

Please be aware that information on this page may be incorrect.

This page describes how I set up a web server and a database server on my Raspberry Pi computer.

For more information about running MX in a Linux environment please see MX on Linux.

For more information about Raspberry Pi computers see Raspberry Pi computer page.

For more information on log files that you might want to convert to database tables, please see MX Weather Diary, Legacy Weather Diary, daily summary log (dayfile.txt), Speciallog.txt, Standard log files, Extra Sensor Files, and Air Link Log.txt.

Introduction

This page records the experience of setting up a web server and database server on my computer.

Most web servers run on Linux, although you can still install them on a Microsoft Windows PC.

It is important to say that many different web servers exist, I have chosen to mention a few, and I have tried various ones. Since “apache2” is very well known, and proabably the most widely used server, I will mostly focus on that. Sometimes, you might choose a simple server that is what is described as light, it uses the minimum resources, but also probably offers the least functionality. In contrast, you might choose apache, because it offers the most functionality, and because it is used so widely there are many people who can help you.

Web server and database server built into Cumulus MX

Web Server built into Cumulus MX

The Admin Interface is a set of web pages that allow you to look at your weather data and change settings.

The MX code will create the necessary web server, it uses the Internet Protocol (ip) address of the computer that is running MX and a port (:8998 by default, but MX has an optional parameter to make it use a different port). If you are viewing the interface on the same computer as runs MX, “localhost” can replace the ip address.

As an aside, there are two different ways to specify ip addresses (IPv4 and IPv6), but this is not place to explain the difference.

Databases built into Cumulus MX

You do not need to know what is in these databases to use MX. I mention them here to explain MX has a built in database server.

Cumulus MX includes two SQLite databases;

  1. The first database was added by Steve Loft, but he never documented what the first table in that database was to be used for (see Cumulusmx.db). The second table in that database is used for Recent history.
  2. The second database is the Weather Diary documented here added by Mark Crossley. Also see in the support forum for a topic comparing the differences between Cumulus 1 and Cumulus MX weather diaries.
    • The Admin Interface provides a page to view, or edit, a single date in the weather diary. You can install another tool that edits SQLite databases if you want (see next section).

The legacy Cumulus software used a XML file to store the weather diary.

Reading content of SQLite databases

Obviously, Cumulus MX can read and update both of these databases, but there are a number of software packages available that can be used outside MX to access these database files, and to minimise technical detail, two will be named as possible candidates, you will find others online.

On Microsoft Windows, install phpLiteAdmin (the significant part of that name is in the middle) to read (you can display the whole table very easily) and edit (via a friendly GUI) either, or both, of the SQLite databases that MX uses as mentioned above.

In Linux use sudo apt install sqlitebrowser to install easy-to-use software to read/edit these databases.

You may need to change file permissions.

Note that sqlite databases use sequential file structure, not a relational database design. This implies for updates, rows are physically identified by row number (order of creation), (not by the primary key that SQL accesses use with relational database designs).

Adding a web server and a database server for your own purposes

MX has (optional) functionality that can upload to a web server, it also has (optional) functionality that can update one or more database tables on a database server.

Frequently, people will use an external web server. This might be one of the Wunderground, Windy, Awekas, WeatherCloud, PWSweathe, WOW, APRS, OpenWeatherMap, and WindGuru external sites that MX can automatically update to. It might be you have subscribed to a commercial web server (and optionally database server). that an external provider supplies to you to run either the 3.10.1 onwards default web site, or it might be for pages you or a third party have designed.

If you have a computer that is left running all the time without consuming a lot of electricity), you might want to add a web server and database server to that device. For example, see Raspberry_Pi_computer_page for a computer with low power consumption that a number of Cumulus users have used for their web server and database server.

As with a commercial web server, you can use your own web server to run the default web pages available in the MX download package. Alternatively, you might want this option, if you are creating your own web pages, and want to try them out without exposure to the public over the internet.

You might select this option to save the subscription costs of a commercial web server (and optionally database server).

All in one versus separate components

To get maximum benefit when using with MX, you need multiple components:

  1. web server – there are various options listed below; beware "apache2" is huge as it is very complicated, but secure; other options consume fewer resources, and may be easier to use, but they will also be more restrictive
  2. PHP script - from [1] - this provides a way of controlling what SQL is run depending on other factors
  3. Database server - MySQL Community edition from [mysql.com] or MariaSQL from [2].
  4. Integrating components like ‘’’phpMyAdmin’’ (there are alternatives) from [3] which combines PHP and MySQL

There are a number of packages that combine all of these:

  • If designed for Microsoft Windows and including Apache2, PHP, and either MariaSQL or MySQL, the package is often described as ‘’’WAMP’’’
  • If designed for UNIX (including Linux), then XAMP will include Apache2, PHP, and either MariaSQL or MySQL

Obviously, there are other web servers, some come on their own, others come as a package.

I will give a few examples, but there are many more, each has its supporters, but also each has people who criticise it. You can search on-line and find both all the (good and bad) comments, and clear instructions.

This Wiki page will only describe one installation (on a Raspberry Pi) in any detail, otherwise it will become far too complex.

Install a web server

I will first mention 3 possible web servers than can run on a Microsoft Windows personal computer (or server box):

  1. You might choose Apache 2 as it is probably the most comprehensive, so if you have enough space on your PC, install it following instructions at the official site [4] (or one of many other web sites that give instructions for installing apache2 on Microsoft Windows).
  2. You might choose ‘’’Uniform Server’’’, which is extremely simple as it comes as a package including PHP script, and MariaSQL database in one package from [5], although you may need to update to a newer release of PHP from [6] and a newer phpMyAdmin from [7] as Uniform Server tends to issue upgrades very rarely.
  3. You might choose to install the Abyss web server from [aprelium.com]

If you get components separately, then there is extra work to get them to work together, but you will find clear explanations on download sites. Generally, if you have an all-in-one package, it seems easier; and normally upgrading one component is also easy (providing its a minor version upgrade, it might be more complex with a major version upgrade).

I will next mention 3 possible web servers than can run on a Linux computer:

  1. You might choose nginx, as it is a quite popular web server for small computers (like the RPi), install this web server with sudo apt-get install nginx -y.
  2. You might choose Apache 2 as it is probably the most comprehensive, so if you have enough space, install it using sudo apt install apache2 -y.
    • You might want to add Fast CGI to your apache server (this is a way of interfacing interactive programmes with a web server, but if you don’t have any idea what that might mean, skip this)
    • To add this use sudo apt install libapache2-mod-fcgid.
  3. You might choose lighttpd, as a web server that is designed to use as little space on your Linux computer as possible, install it with sudo apt-get install lighttpd -y.

For a Raspberry Pi, installing two of these is covered at [8]

Install PHP Hypertext Pre-processor

As already mentioned, some web servers come in a package that includes this script language. Again, as mentioned before, such packages may be rarely upgraded so the included PHP might be an old version with bugs or security concerns.

At time of updating this Wiki page, PHP 8.1 is stable version, PHP 8.2 is available to test, and PHP 9.0 is planned. Older versions 7.x, and 5.x are largely obsolete, although some web providers may still make them available.

  • PHP is not the only script language available, but it is quite comprehensive
    • PHP can be used either in a fairly simple way by non-technical people
    • or in an object-oriented way for those more technical to achieve success with more complex scripts.
  • To check which PHP modules have been installed by any command, type php -m.
  • To test that php is installed, type php -v into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.
  • If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least mysqli and mbstring.



Microsoft Windows - download/install PHP

Straight forward, just down load a zip from from [9]. You will want a “multi-threaded” version to use with a web server, as several PHP web pages may need to run at the same time.

  • PHP is not the only script language available, but it is quite comprehensive
    • being able to be used either in a fairly simple way by non-technical people
    • or in an object-oriented way for those more technical to achieve success with more complex scripts.
  • To check which PHP modules have been installed by the above command, type php -m.
  • If you later want to use a database (and a tool like PhpMyAdmin), then your php modules loaded must include at least mysqli and mbstring.


To test that php is installed, type php -v into terminal, and you will see the exact version that has been installed, a confirmation that it offers command line interface (cli), and a copyright notice.

Linux operating systems - best repository

The default repositories (see here) may not contain up-to-date versions of all software, or may contain some software that is not compatible with other software.

You can add another repository that generally has reasonably new PHP versions (a few weeks behind the official [10] site) using

sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt update


  • On Linux, the simplest instruction to install it is sudo apt install php -y, although in practice this will not install all the PHP modules you actually need
    • A warning, which version of PHP that command gives you depends on which php version is found first in the respositories your computer has in its source list, so you may end up with the oldest!
  • A better variation is to actually list the version we want:

sudo apt install phpV.x (where V is major version, and x is minor version e.g. 7.4 or 8.1)

    • This may still not give us the modules we want, but it does control which version is obtained
  • Therefore the best variation is to list both the modules we want, and the version we want (here "x" denotes minor version, adjust to what is latest stable (e.g. change to php8.1-common etc.):

sudo apt install -y php8.x-common php8.x-cli php8.x-json php8.x-xml php8.x-readline php8.x-mbstring php8.xmysql php8.x-curl php8.x-gd pdp8.x-psqlite php8.x-pdo -y

  • The Linux package manager will ensure that any components, that are required by dependencies, but not listed specifically, are also installed
    • Loading from the "packages.sury.org/php" repository has one disadvantage, the dependencies there don't automatically integrate the PHP with your web server (if you are using apache 2 for web server, add "libapache2-mod-php8.x" to list);

Linux operating systems - standard repository

  • On Linux, the simplest instruction to install PHP is sudo apt install php -y, although in practice this will not install all the PHP modules you actually need
    • A warning, which version of PHP that command gives you depends on which php version is found first in the respositories your computer has in its source list, so you may end up with the oldest!
  • A better variation is to actually list the version we want:

sudo apt install php7.4

    • This still does not give us the modules we want, but it does control which version is obtained
  • A still better variation is to list the components we want:

sudo apt install php7.4 php7.4-cli php7.4-common php7.4-json php7.4-xml php7.4-opcache php7.4-readline php7.4-mbstring php7.4mysql php7.4-curl php7.4-gd php7.4-sqlite php7.4-pdo -y

    • Add ”libapache2-mod-php7.4” as just part of what integrates apache 2 and PHP
    • The included “php7.4-cli” is what allows us to run scripts in batch
    • “php7.4-common” and “php7.4-mbstring” are pre-requisites for various other modules, and the second is used by ‘’’phpMyAdmin’’’ and other tools
    • “php7.4-json” (see here and Category:JSON_Files#Data_Transfer_Format and “php7.4-xml” (see this section) are used to handle particular file types
    • “php7.4mysql” integrates php and MariaSQL/MySQL
    • "php7.4-sqlite php7.4-pdo" are needed to access the weather diary or recent history databases
    • “php7.4-curl” “php7.4-gd” may be needed as these are used by some third-party routines designed by Cumulus contributors
  • There are advantages in using default repositories:
    • installing a web server and PHP from the default repositories will pick up joint dependencies and thus integrate the web server and php.

Creating a Home web page on your web server

You may wish to create a index.php web page at (Linux Apache server) /var/www/html which is the web server root for browsing, or of course you may wish to copy or FTP here all your existing web pages. To view a php web page, go into a browser and type in a url with the same IPv4 address as you use for the admin interface, omit the port (:8998) and instead type in your web page name (e.g. //192.168.1.xy/index.php, where you need to determine digits that replace xy).

Finding your PHP configuration file

For sake of simplicity in this article, from now on I will assume the web server you installed is "apache2", change that segment (in the paths quoted below here), if you installed a different web server.

The web server main php.ini is found at /etc/php/m.n/apache2/php.ini (where 'm.n' depends on which PHP version was installed, that was found out in last sub-section).

You may need to edit this file for example to specify where your include files are stored (if not in same folder as script with require/include). Replace "apache2" by "cli" in the path for the batch php.ini file that you may also need to edit.

To run a php script in batch, type php - f <file_name>. You can redirect the output by adding a greater than symbol and the destination file (i.e. > log_file) on the same terminal line.

If you want the MX external commands to run a PHP script for you, use something like "sh" as the program to run (i.e. run shell script); and in the parameters something like /home/pi/CumulusMX/MXutils/autoEOD.sh will run a script "autoEOD.sh" you have added to the "MXutils" folder. In that script, you put something like (assuming you have added a folder 'batch' with a subfolder 'log'):

#!/bin/sh
# This MX batch command file is initiated automatically by Cumulus MX software during last stage of processing the end of a meteorological day
echo "It stores feedback in log file CumulusMXDailyBatch (file name ends with day of month)"
sudo php -f the_path_and_filename_goes_here.php > /home/pi/CumulusMX/batch/log/DailyBatch_Day$(date +%d).log


Install Maria database

MySQL database software is controlled by Oracle and not made available for inclusion in Raspberry Pi repository.

MariaSQL is an alternative that has largely similar command syntax so is likely to work with code (like MX) designed to work with MySQL. Since the MX developer (Mark Crossley) actually uses Maria DB, we can have plenty of confidence it is suitable.

To install this database server, we type sudo apt install mariadb-server php-mysql -y.

Our database files will be stored at (Linux) /var/lib/mysql by default. Our MariaDB configuration is stored at /etc/mysql/mariadb.conf.d/50-server.cnf, and it is the datadir= entry that controls where the database files are stored, so edit that to store files somewhere else, but make sure the storage location has right ownership permissions. This Wiki page will not give you the full technical details, so only those with some technical knowledge do such changes.

Making your database secure

We need to assign passwords to control access to the database by typing into terminal sudo mysql_secure_installation. That brings up a screen:

  1. where we are asked to type current password for the root (as no password has yet been set, simply press Enter),
  2. next type Y to signify we are going to set a new password for root,
  3. next type in a new password that you will not forget,
  4. next as explained on the screen we are going to say whether users must select a user name as well as a password, type Y to Remove anonymous users
  5. next we have to decide whether we will only be logging into the database on our Pi (using Localhost) or we might be logging in remotely; type Y or N respectively, but if you choose N remember someone other than you might guess there is a root user and might guess the password you set,
  6. next we have another option of whether to retain or delete a test database, answer N or Y respectively, I would keep the test database for now as you can play with it and then remove it later,
  7. finally you type in another Y as that will Reload the privilege tables and ensure all is set up for your access to the database later.

The message, when the process successfully completes, is "Thanks for using MariDB".


Getting web and database servers ready for use

We need to create a user for PhpMyAdmin (or adMiner or whatever) to access our database and another for Cumulus to use to access the database tables. At the moment our database access has the single root@localhost user we created when we installed MariaDB. The initial password was set then, and we need to use it to get access to MariDB monitor where we can insert some SQL commands to create these two users.

Most tools, like PhpMyAdmin, on first start up will ask for username (here I choose "admin") and password, thereafter the tool should use same log-in (by default you see log-in screen each time you restart or if it is left idle for a long time), let us create a user called 'admin' for it.

The database name (here I choose "cumulus" for the data base name), user name (here I choose "weather" for the user name), and password, must all match those set in MX using the MySQL_settings in the admin interface. You could of course use PhpMyAdmin (or AdMiner) to create additional user names, and to create the database, but I assume all is done in the following bit of SQL. Remember, the Windows operating system is not case sensitive, but all Linux based operating systems are case sensitive (so whatever pattern of capitals and lowercase you choose must be used every time. Also all names must start with a letter, can contain only letters or digits (no punctuation), and must not be a reserved word ("password" is a reserved word, so you cannot use that for a password, nor for a column name).

Obviously, these names might not be what you use, but you can amend commands below accordingly. For each line with SQL, it must end with a semicolon (;) as shown. After you press "Enter" key you will get a response saying "OK" if you have remembered the semi-colon. You can actually use "\G" or "\g" instead of a semi-colon, but here we will keep it simple and stick to semi-colon.

The SQL lines have a prompt of a greater than symbol (>) while the command lines have a prompt showing current path. Note that "identified by" is followed by a password enclosed in single quotes.

sudo mysql --user=root --password=InitialPassword
create user admin@localhost identified by 'PhpMyAdminPassword';
create user weather@localhost identified by 'MXPassword';
grant all privileges on *.* to admin@localhost;
grant all privileges on *.* to weather@localhost;
FLUSH PRIVILEGES;
create database cumulus;
exit;

As I type this, Cumulus MX has no exception handling if the username and password defined in the settings do not exist in the database, therefore in this situation it will crash out (with message press Enter to close).


Installing Adminer, or PhpMyAdmin

ExportMySQL.exe and CumulusMX.exe (see Cumulus_MX#Executables for details) both create SQL for updating tables in a MySQL database, such as the one our MariaDB software package we have installed can create.

This option is about installing a package to do the tasks that go with operating a database, that MX may not be able to do in the way you require.

PhpMyAdmin

PhpMyAdmin is one tool that can be used to manipulate your MySQL like database (that is the significance of the "My" bit in the middle of the tools's name).

You may like PhpMyAdmin as this offers:

  • a graphical approach (you see a table on screen and navigate to the row or cell you want to work on)
  • a SQL approach (you can try out any SQL here, before adding it to a script that you might use in a web page)
  • a selection approach (you select a database, then select a table, then select an action)

WINDOWS:

  1. Download from official download page
  2. You need to do some configuration (PLEASE EXPAND THIS).

LINUX:

  1. The version of phpmyadmin in the default repository is not compatible with PHP7.2 and above
  2. Visit the phpMyAdmin official download page and look for the .tar.gz URL and download it using wget.
  3. The following steps assume version 5.2.0, released in 2022 is to be downloaded. (If a later version is now available, make sure to change the commands below to match)
  4. Decide whether you just want English, or need another language
  5. Change to folder where we will extract our download, this could be cd /usr/share/ or some other directory perhaps on an extra storage drive
    • For English only, extract your download into the current folder using sudo tar xzf phpMyAdmin-5.2.0-english.tar.gz
    • For other languages, extract your download into a sub-folder of the current folder using sudo tar xzf phpMyAdmin-5.2.0-all-languages.tar.gz
  6. Create a folder to hold the download within the folder where we did our download sudo mkdir phpmyadmin_5.2.0
  7. Now we will move the contents from the sub-folder created by the "tar" command into the folder we just created
    • For English only, the move command is sudo mv phpMyAdmin-5.2.0-english/* ./phpmyadmin_5.2.0
    • For other languages, the move command is sudo mv phpMyAdmin-5.2.0-all-languages/* ./phpmyadmin_5.2.0
  8. There is some configuration to do, for simplicity here assume the destination directory ("./phpmyadmin_5.2.0") is actually "/usr/share/phpmyadmin" and that we will use '/var/lib/' and '/etc/' to store other directories used by phpmyadmin:
    • First sudo nano /usr/share/phpmyadmin/libraries/vendor_config.php to edit (with nano) the vendor configuration
      • change the path shown in the existing line mentioning 'TEMP_DIR' so it becomes define('TEMP_DIR', '/var/lib/phpmyadmin_5.2.0/tmp/');
      • change the path shown in the existing line mentioning 'CONFIG_DIR' so it becomes define('CONFIG_DIR', '/etc/phpmyadmin/');
  9. The following sequence of commands will
    • add the mysqli module to our php install, sudo phpenmod mysqli
    • will restart apache (change if you are using a different web server), sudo service apache2 restart
    • will create a symbolic link for the phpadmin installation in the server web root so it can be seen (and used) in our browser, sudo ln -s /usr/share/phpmyadmin /var/www/html/phpmyadmin, note the first path is to directory where we stored our phpmyadmin files, and the second path/file is the default web directory and the symbolic link file name, adjust these paths to what is needed for you.
    • will give the standard user (pi) ownership of the database files and the web pages: sudo chown -R pi:www-data /var/www/html/
  10. phpMyAdmin now has the files where it expects to see them, and should work
    • To log in requires a username (default "admin") and password (defined in your database set-up - see previous section)
    • The remaining configuration can be done within phpMyAdmin, follow cues on each screen as work through all configuration tabs.


If you have any difficulty, there are some instructions at https://devanswers.co/manually-upgrade-phpmyadmin/ and some at https://docs.phpmyadmin.net/en/latest/setup.html#quick-install


adminer

You may prefer to install a different tool, perhaps adminer that works using a drill down approach. A drill down approach is when you select the database, then select the table, then select the row, then select the column, then select the action. This logical step by step approach is a popular approach, but does not suit everyone.

Install this drill down package with sudo apt install adminer (I leave you to work out the commands needed after that).


Viewing web pages on our new web server

You can view any index.php or PHPMyAdmin web page in your browser by prefixing the address with your Pi IPv4 address e.g. a URL like http://192.168.1.xyz/phpmyadmin where xyz is 2 or 3 digits you look up as mentioned before where FTP was described. If your Pi has both Ethernet and wireless connection, there will be two different values of xyz for you to choose one from. On first run of PhpMyAdmin, as already mentioned, you will see a PHP MyAdmin log-on page where you type username and password we have just set up.

Populating your database tables on your Pi

Cumulus MX has functionality to update database tables at one of several intervals:

  1. fixed SQL at real-time
  2. fixed SQL at standard interval
  3. fixed or custom SQL at end of day
  4. custom number of seconds
  5. custom number of minutes

MX now has ability to run SQL updates when you edit a log file, provided you use the predefined column and table names. MX also has the ability to store failed SQL and to try to run it again at the next rollover. This will fail again and again until MX is stopped, if the SQL is specified wrongly, but is designed to cope with correct SQL, that fails because the database server is down, and assumes the database server may be working again by next rollover.

The database tables can use the column names in the schema pre-defined by Cumulus MX for the predefined queries, or in a custom schema (where you specify the column names) if you specify the required SQL. The settings are all in SQL settings, so read that cross-reference to find out more.

You might have started using MX before you set up your database. There is a option in that part of the admin interface to create database tables (as required) for each of the first 3 updating intervals. For example, the default name for the table updated at the standard interval is "monthly", but you could give it a name of "standard" or whatever you like.

There is an optional utility that can be installed. Type cd CumulusMX && sudo mono ExportToMySQL.exe parameter to run the executable. The mandatory parameter specifies the log file name to read (dayfile.txt or standard monthly log file).

  1. for each line in the log file the executable will try to insert a row in the database table
    • the SQL syntax used is "insert ignore", so if the row for that log file line already exists in the table, it will skip onto next line of log file.

There is no way to use this executable for insert of past rows into the real-time table.

If you have been running Cumulus on another device previously and already have database tables, the next section explains how you can create SQL to export your old database and use that SQL to populate the new table.

Transferring database tables to your Pi

If you have been using Cumulus before (and already have a database) then you can use PhpMyAdmin on your old device to export out all the Cumulus tables as SQL in a zip file, FTP that zip file across to your Pi, then use PhpMyAdmin on your Pi to import that zip file.

Providing you selected the right options for what SQL you created in your export, the import will contain SQL to create the tables and to insert all the rows into each Cumulus table on your Pi. Please note that there is a limit of around 1000 rows that can be imported in one action, so for your bigger tables, you will only create the table once, but you will do several Replace row actions (export from old device, then import to Pi) each transferring just one thousand rows, until the whole table is on the Pi. You may prefer to use "ExportMySQL.exe" as described in previous section to recreate your bigger tables from the Cumulus log files.

You might want to also export/import the tables in the PhpMyAdmin database (as they contain your preferences for each of your tables) in a separate zip, although these might need some extra transformations, as they are specific to a particular version of the database server (and the old database server version may not match the MariaDB version on your Pi). Some PhpMyAdmin tables do change for different versions of the tool, so that too may make export/import of its tables more complicated.

Restarting Web Server

After all these installs, we need to restart your web server (so it loads the PHP and MariaDB), if you have installed apache 2 do this by typing sudo service apache2 restart or (if we just want the Apache configuration reloaded) by typing sudo systemctl reload apache2. Similar commands apply for other web servers.

We will need to restart Apache (or whichever web server we installed) any time we change our php.ini files, database passwords, and anything else that is checked when the web server starts.