Difference between revisions of "Synchronisation of data between NASs"

From LinnDocs
Jump to: navigation, search
(Created page with 'This describes the means of synchronising data between several Mac mini NASs. We have a single ''master'' NAS on the main plant network, and 3 NASs in the South Wing area, one f…')
 
(→‎Configure SSH access from Clients)
 
(11 intermediate revisions by one other user not shown)
Line 8: Line 8:
 
Due to the network topology, the isolated networks can access the main plant network, but access the other way round is not available (without some non-default reconfiguration of network components). Hence a ''pull'' synchronisation method was required, so that changes from the ''master'' NAS are pulled in by the ''slave'' NASs in a regular manner.
 
Due to the network topology, the isolated networks can access the main plant network, but access the other way round is not available (without some non-default reconfiguration of network components). Hence a ''pull'' synchronisation method was required, so that changes from the ''master'' NAS are pulled in by the ''slave'' NASs in a regular manner.
  
−
This has been achieved using the standard OS/X (and Unix) tools '''rsync''', '''cron''' and '''ssh'''. As this uses an SSH connection the synchronisation method described below should be extensible to any setup where the NASs can be connected using SSH, including remotely across the internet (although bandwidth requirements are likely to be a major factor in this case).
+
This has been achieved using the standard OS/X tools '''rsync''', '''launchctl''' and '''ssh'''. As this uses an SSH connection the synchronisation method described below should be extensible to any setup where the NASs can be connected using SSH, including remotely across the internet (although bandwidth requirements are likely to be a major factor in this case).
 
   
 
   
  
Line 20: Line 20:
 
=== Configure SSH access from Clients ===
 
=== Configure SSH access from Clients ===
  
−
For every '''client''' ''(slave)'', it is necessary to configure SSH access to the server. The method described here includes steps to remove the requirement for a password to be entered (so that the synchronisation can be executed automatically). As described this leaves a '''security hole''' in that anyone with access to the client machine can access the server machine without any further restrictions.
+
For every '''client''' ''(slave)'', it is necessary to configure SSH access to the server. The method described here includes steps to remove the requirement for a password to be entered (so that the synchronisation can be executed automatically.) As described this leaves a '''security hole''' in that anyone with access to the client machine can access the server machine without any further restrictions.
  
 
On client
 
On client
−
* Open terminal window  and execute ''ssh-keygen -t dsa''
+
* Open terminal window  and execute ''ssh-keygen -t rsa''
 
** Accept defaults for all prompts (file location and blank passkey)
 
** Accept defaults for all prompts (file location and blank passkey)
−
* Open the file ''~/.ssh/id_dsa.pub'' and copy the entire (one line) contents
+
* Open the file ''~/.ssh/id_rsa.pub'' and copy the entire (one line) contents
 
* Open another terminal window and connect to the server
 
* Open another terminal window and connect to the server
 
** ''ssh <user>@<address>''
 
** ''ssh <user>@<address>''
Line 31: Line 31:
 
** Enter password for server to connect
 
** Enter password for server to connect
 
* In the server window create (or open) the file ~/.ssh/authorized_keys
 
* In the server window create (or open) the file ~/.ssh/authorized_keys
−
* Append the contents of the clients id_dsa.pub to this file and save it
+
* Append the contents of the clients id_rsa.pub to this file and save it
 
* Exit the server, and try reconnecting
 
* Exit the server, and try reconnecting
 
** ''ssh <user>@<address>''
 
** ''ssh <user>@<address>''
 
** This time should connect without requiirement to enter password
 
** This time should connect without requiirement to enter password
 +
 +
Note: macOS Sierra uses OpenSSH 7.0 which deprecates the use of DSA keys.
 +
It's now recommended to use RSA, so the above has been edited accordingly.
 +
 +
=== Configure timed auto-execute of rsync on Clients ===
 +
 +
The rsync command line which is required (to sync between 2 USB drives both called USBDrive1) is
 +
rsync --quiet --update --recursive --whole-file --delete-during <user>@<address>:/Volumes/USBDrive1/Music /Voumes/USBDrive1
 +
 +
This can be made to auto-execute at regular time intervals using '''launchctl''' on Mac
 +
* On every '''client
 +
** Create the ''plist'' file '''~/Library/LaunchAgents/com.user.sync.plist'''
 +
** Ensure this is created under the '''same user''' that the SSH keys setup above
 +
** Update '''name''' and '''address''' in the file below
 +
** This gives a repetition interval of 1/2 hour (1800 secs) as defined at end
 +
<?xml version="1.0" encoding="UTF-8"?>
 +
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
 +
<plist version="1.0">
 +
<dict>
 +
    <key>Label</key>
 +
    <string>com.user.sync</string>
 +
    <key>ProgramArguments</key>
 +
    <array>
 +
        <string>/usr/bin/rsync</string>
 +
        <string>--update</string>
 +
        <string>--recursive</string>
 +
        <string>--whole-file</string>
 +
        <string>--delete-during</string>
 +
        <string>--verbose</string>
 +
        <string>name@address:/Volumes/USBDrive1/Music</string>
 +
        <string>/Volumes/USBDrive1</string>
 +
    </array>
 +
    <key>StartInterval</key>
 +
    <integer>1800</integer>
 +
</dict>
 +
</plist>
 +
 +
* Add this file to the launchctl executor
 +
launchctl load ~/Library/LaunchAgents/com.user.sync.plist
 +
 +
 +
To test:
 +
launchctl start com.user.sync
 +
 +
 +
Any time changes are made to the plist file, launchctl must be reloaded
 +
launchctl unload ~/Library/LaunchAgents/com.user.sync.plist
 +
<update plist>
 +
launchctl load ~/Library/LaunchAgents/com.user.sync.plist

Latest revision as of 15:08, 3 November 2016

This describes the means of synchronising data between several Mac mini NASs.

We have a single master NAS on the main plant network, and 3 NASs in the South Wing area, one for each of the isolated networks.

  • Linn Home
  • Dem Rooms
  • Welcome Area

Due to the network topology, the isolated networks can access the main plant network, but access the other way round is not available (without some non-default reconfiguration of network components). Hence a pull synchronisation method was required, so that changes from the master NAS are pulled in by the slave NASs in a regular manner.

This has been achieved using the standard OS/X tools rsync, launchctl and ssh. As this uses an SSH connection the synchronisation method described below should be extensible to any setup where the NASs can be connected using SSH, including remotely across the internet (although bandwidth requirements are likely to be a major factor in this case).


Configure Remote Access on Server

On the server (master), remote (ssh) access must be enabled

  • System Preferences -> Internet & Wireless -> Sharing
  • Check the Remote Login service


Configure SSH access from Clients

For every client (slave), it is necessary to configure SSH access to the server. The method described here includes steps to remove the requirement for a password to be entered (so that the synchronisation can be executed automatically.) As described this leaves a security hole in that anyone with access to the client machine can access the server machine without any further restrictions.

On client

  • Open terminal window and execute ssh-keygen -t rsa
    • Accept defaults for all prompts (file location and blank passkey)
  • Open the file ~/.ssh/id_rsa.pub and copy the entire (one line) contents
  • Open another terminal window and connect to the server
    • ssh <user>@<address>
    • Answer yes if prompted to accept new connection
    • Enter password for server to connect
  • In the server window create (or open) the file ~/.ssh/authorized_keys
  • Append the contents of the clients id_rsa.pub to this file and save it
  • Exit the server, and try reconnecting
    • ssh <user>@<address>
    • This time should connect without requiirement to enter password

Note: macOS Sierra uses OpenSSH 7.0 which deprecates the use of DSA keys. It's now recommended to use RSA, so the above has been edited accordingly.

Configure timed auto-execute of rsync on Clients

The rsync command line which is required (to sync between 2 USB drives both called USBDrive1) is 

rsync --quiet --update --recursive --whole-file --delete-during <user>@<address>:/Volumes/USBDrive1/Music /Voumes/USBDrive1

This can be made to auto-execute at regular time intervals using launchctl on Mac

  • On every client
    • Create the plist file ~/Library/LaunchAgents/com.user.sync.plist
    • Ensure this is created under the same user that the SSH keys setup above
    • Update name and address in the file below
    • This gives a repetition interval of 1/2 hour (1800 secs) as defined at end
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> 
<plist version="1.0">
<dict>
    <key>Label</key>
    <string>com.user.sync</string>
    <key>ProgramArguments</key>
    <array>
        <string>/usr/bin/rsync</string>
        <string>--update</string>
        <string>--recursive</string>
        <string>--whole-file</string>
        <string>--delete-during</string>
        <string>--verbose</string>
        <string>name@address:/Volumes/USBDrive1/Music</string>
        <string>/Volumes/USBDrive1</string>
    </array>
    <key>StartInterval</key>
    <integer>1800</integer>
</dict>
</plist>
  • Add this file to the launchctl executor
launchctl load ~/Library/LaunchAgents/com.user.sync.plist


To test: 

launchctl start com.user.sync


Any time changes are made to the plist file, launchctl must be reloaded 

launchctl unload ~/Library/LaunchAgents/com.user.sync.plist
<update plist>
launchctl load ~/Library/LaunchAgents/com.user.sync.plist
Retrieved from "https://docs.linn.co.uk/wiki/index.php?title=Synchronisation_of_data_between_NASs&oldid=13031"