PHP Uploads

What is it?

In March 2023 for Cumulus MX version 3.24.0 a new PHP upload mechanism was introduced as an alternative to using FTP, FTPS, or SFTP (FTP over SSH).

It uses the HTTPS or HTTP protocol for transferring file data. It requires that you have a PHP enabled web server (most are), and that you upload a small PHP file to the root folder of your web site.

Cumulus MX then communicates with that script to transfer the data, and the script saves the data to a file on your web site.

It can transfer both text and binary files.

If the upload.php file is installed for CMX and the protocol works for CMX then it will also work for CUtils and the file locations will be relative to the location of the upload.php file (which therefore will be the root location of your website).

Note: It is not a requirement to change to using the PHP Upload method. The previous FTP/FTPS/SFTP methods remain fully supported and operational.

Is it secure?

Yes! It uses a shared secret generated by MX (or you can enter your own). This secret is entered into the PHP upload file before you upload it to your web site.

The secret is used by MX to create an encrypted hash of the upload. The upload script on your server generates its own hash and compares it with the one sent by MX with the data. If they do not match the upload is not allowed.

In addition to this, timestamps are used to ensure that an upload cannot be repeated by replaying the same upload.

Because no sensitive information is sent in the request, the PHP upload mechanism can be safely used with the plain old HTTP protocol

What are the advantages?

• It is fast!

• It is much faster than using FTP(S).
• It uses gzip compression of the data across the network (web server support required)

• It is reliable

• If your web site is up then it should work, it removes the reliance on your web providers FTP server working.

• It reduces bandwidth.

• Unlike FTP which uploads the whole file each time, the PHP upload can send incremental data. For the default web site, most the data is the graph files and every update each entire file is sent over FTP every time just to add one new data point. Using the PHP upload, just that new data point is sent for each file, and the PHP script appends it to the existing file on your server.
• If any updates fail, then only the cumulative data since the last successful transfer is sent.
• It uses data compression over the network

Disadvantages?

• It requires PHP support from your web provider - though most do

• Some web providers may not permit all the PHP methods that are required to implement this method

• You need to change your upload paths to use the new method - make a copy of your cumumulus.ini file so you can easily revert

Configuration

1. Run CMX and change the Upload protocol to PHP, set the URL to your web site + upload.php
   for example: https://mysite.com/weather/upload.php
2. Copy the suggested unique "PHP Secret" from the Internet Settings (or make up your own)
3. Edit the /webfiles/upload.php file, and change the default secret change_this_to_the_value_in_CMX to the value in CMX
4. Copy the /webfiles/upload.php to your web server, place it in the root folder of the default web site along with the index.html files etc
5. You will need to update the paths for the NOAA Reports (it probably needs to be just “Reports”)
6. Amend the paths of any Extra Files you upload (it probably needs to be just the remote file name)
7. The uploads should start working!
8. You can now disable all the "Create local" options for files in Internet Settings|Interval Settings. Local copies of the files created in the /web folder are no longer required

It is important to understand the relative paths used by upload.php as this is quite different from FTP configurations where sometimes full paths are required (SFTP) or paths from the FTP root (which may differ when full ownership or shared hosting). All files will be transferred relative to that location. You can go deeper and sideways into the directory tree but not up. See below**

Migrating your FTP settings

When setting the Upload protocol to PHP you may need to make the following upload settings relative as well:

1. NOAA Reports
2. Extra Web Files
3. The Moon image

Other:

1. Disable 'Create local' copy unless required (Windspeed and Wind direction are required local for CumulusUtils)
2. Logging for PHP upload is found in the standard MXdiags logfile, FTP logging has no meaning when using PHP

NOTE: When using the Wizard you will also encounter Upload configuration which obeys the same rules. NOTE: The settings are valid not only when you use CMX but also for CumulusUtils which does not have any setting for upload.PHP of its own!

**Extending the upload path scope

By default, the upload script only allows you to upload files to the same directory as the upload.php script or deeper. However, it is possible to relax this restriction. In the upload.php script there is a variable...

$limitPath = true;

Changing this to false means the script will be allowed to create files anywhere in server file system that the PHP process has write permissions.

This option should only be used if you are certain, you need it, and ideally you can restrict the PHP process file system permissions to the areas you want data written.

CumulusUtils

The CUtils inifile parameter UploadDir specifies a subdirectory for CUtils relative to the webroot (the location of upload.php) without any specification of the URL (as that is implicit in the use of upload.PHP as specified in Cumulus.ini). If you wish all files to be transferred into the same directory as specified by CMX, then leave UploadDir empty.

Cumulus.ini

The following settings in CMX Cumulus.ini [FTP site] section control the PHP upload configuration.

These entries would normally be set via the Internet Settings configuration screen.

The upload.php procedure file can be copied to any location you wish but the following CMX ini file parameters determine its operation and need to reflect its true location and CMX configuration (in case you are running more than one instance):

PHP-URL=<URL>
PHP-Secret=<key as configured in CMX>

Potential Issues - Technical!

File Permissions

On my Linux self-hosting server, I found I had not given the PHP process write permissions on the folders holding my web site. What I did is (and this may not be the best or only method)...

1. Create a new user group - webwriters - and add my existing user 'pi' and 'www-data' (the user PHP and the web server run under) to that group:

	> sudo addgroup webwriters
	> sudo usermod -a -G webwriters www-data
	> sudo usermod -a -G webwriters pi

2. Restart the web server and PHP-FPM to pick up the new group membership

	> sudo service nginx restart
	> sudo service phpN.N-fpm restart

3. Then changed the group associated with the web folder(s)

	> sudo chown -R :webwriters /var/www/mywebsite

4. Finally change the permissions for webwriters group on the folder

	> sudo chmod -R g+rwx /var/www/mywebsite
	> sudo chmod -R o-wx,o+r /var/www/mywebsite

You can check what user PHP is running under with a simple script...

<?php echo `whoami`; ?>

Or, you can use the "top" or "htop" command, and look for the service there - check your web server too!