[[other_stuff:remote_sync]]

Updated: Version 4.0.3

22.03. Remote synchronisation

mSupply's remote synchronisation allows mSupply running stores on local servers to send data to a central mSupply server for combined reporting.

It's like this:

  • The primary server is a central mSupply server, which controls:
    • the master system data which is common across all stores e.g. names, items and their visibility
    • the store records themselves, and their non-sync related preferences (⇒ all stores exist on the primary)
  • The sync server is a central mSupply server (often running on the cloud) which:
    • aggregates the store data from all stores (⇒ centralised reporting, and a real-time backup)
    • controls how all of the stores sync with each other
    • routes transfers from one store to another
    • runs the mSupply dashboard
  • In any given setup, there will only be one sync server, and it must be running a web server. There will also only be one primary server. In many cases, these are actually the same, but since mSupply v4.03 it has been possible to keep them separate - this has some advantages:
    • the primary server doesn't need to be online all the time, so it can be a local server and still be able to centrally administer system data, without being a bottleneck for sync
    • the sync server (especially if it's in the cloud) can be always online and available for remote access to reports and the dashboard, and for processing and routing of sync data
  • A satellite site is any mSupply server (or single-user standalone copy of mSupply, or a copy of mSupply mobile) which is not the primary server i.e. it cannot administer system data (⇒ the sync server, if different from the primary server, can also be a satellite site)
  • A remote site is any mSupply server (or single-user standalone copy of mSupply, or a copy of mSupply mobile) which is not the sync server i.e. it cannot administer sync (⇒ the primary server, if different from the sync server, can also be a remote site)
  • The syncing process is initiated from each remote site on a schedule, and pushes its own store-specific data to the sync server
    • if the remote site is also a satellite, it pulls any system data from the sync server
    • if the remote site is also the primary, it pushes system data to the sync server
  • The primary server, sync server and all of the satellites and remote sites are collectively referred to as sync sites.
  • Each sync site has a unique ID and connection parameters (IP address, username and password), setup in the synchronisation preferences, along with a list of the other sync sites with which it can send/receive synchronisation data. These settings are complicated, easy to get wrong, and so should only be modified by Sustainable Solutions.
  • To preserve data integrity, and to avoid potential clashes where more than one sync site tries to modify the same data, only one sync site can edit/update any specific type of data.
    • System data which is common to all stores (e.g. suppliers, customers, items) can only be edited or imported on the primary server - changes are synced to all satellite servers (see below for details).
    • Store-specific data (e.g. transactions, stock, locations) can only be edited or imported on the sync site where the store is active. Changes to store-specific data are only synced to the instance of the corresponding store on other sync site(s), depending on the store sync preferences (see below for details).

Stores to be synced will exist as separate instances of the same store on more than one sync site:

  • on the primary server - the store itself and its non-sync related preferences are editable ⇒ all stores must exist on the primary server
  • on the sync server - the store's sync preferences are editable ⇒ all stores must exist on the sync server

On a mirrored sync system, the store must be created first of all on the primary server. It will then sync to the sync server, where the store's site can then be assigned and its sync preferences configured.

  • store-specific data (e.g. store transactions, stock) can only be edited on a single sync site - the site on which the store is defined as Active. This is controlled on the sync server by the store preferences on the Synchronisation tab for each virtual store on each sync site:

Synchronisation type drop down list: Shows the sync type of the current store. Will be editable on the sync server if you have unlocked it using the Click to unlock button.

Click to unlock button: Click it to enter the unlocking password to enable you to edit the settings on this tab.

Set as local store checkbox: Will be editable on the sync server if you have unlocked it using the Click to unlock button. If checked, this will change the store's Sync ID to be that of the current site and will uncheck any of the elements in the Local column in the Sync with table.

Include transactions in sync checkbox: Only affects dispensary stores, as transactions are always synced otherwise. If checked for a dispensary store, this will also sync prescriptions and related dispensary data. As this detailed information is rarely needed at central level, and can potentially generate a lot of extra sync traffic, the default is that it is switched off.

Sync ID field: Displays the sync ID of the current store so that you can see which one in the table below you're talking about! The store's sync ID should almost always be the ID of the site where the store is active.

Sync with table: Shows the other sites in your sync setup that the current store will sync with. Also defines what relationship those other store instances have to your current one and therefore what records need to be sent to that site from the instance of the current store on the current site. Will be editable (only on the sync server) if you have entered the password using the Click to unlock button. Checking any checkbox in the Local column will set the store's Sync ID to be that of the selected row, and will uncheck any others as well as the Set as local store checkbox.

This setting is a bit confusing, but it does work to understand it as 'whatever the sync type of the store is on that site'.

If your version of mSupply server is pre 3.50, after changing sync settings for a store you will have to restart the mSupply application on the primary server for the changes to be applied.

  • Once a store has been set up (see the relevant parameters below), item and store visibility for the store needs to be set up on the primary server - see Virtual stores, Controlling item visibility.
  • To do that, once you log in to the primary server, you need to switch to the new store - see Switching store
  • To do that, you need to have permission to log in to it - see Managing users (It's a good thing this doesn't happen often! :-D)

These settings are necessarily complex and should only be modified by Sustainable Solutions, after discussions to agree on the configuration which is the best fit for requirements.

Each store in a synchronised system needs to have a sync “type” as described here.

Active

A store whose sync type is Active means that the store-specific data can be edited (added, edited, deleted) on the sync site where you are editing/viewing this store.

Collector

A store whose sync type is Collector means that the store-specific data can not be edited on this sync site. The store on this server receives all store data from an active store on another sync site. i.e. it is a replica of a corresponding active store on another sync site. This means that the store can be examined and reports generated on this sync site.

Transfer

A store whose sync type is Transfer is similar to the Collector type except that the store on this sync site is used only as a vehicle for receiving stock transfers or requisitions from other stores on the same sync site and passing them on to the corresponding store on other sync sites. It doesn't receive any other store data (such as stock lines). That is, it is not a replica of an active store, and should not be used as such. The store can not be examined or have reports generated on this sync site. The benefit of a Transfer store over a Collector store is that Transfer stores require less data to be synced between servers.

The Store 'sync-with' options control which store-specific synchronisation data gets sent to the other sync sites which are linked to that sync site:

None

A value of None means that this store doesn't sync with the corresponding store on the selected sync site. That is, there is no instance of this store on that sync site.

Active/collector

A value of Active/collector means that store-specific data edits, including any edits to the core store data, are synced to the corresponding store instance on the selected sync site. The store instance on the selected sync site will normally be of type Active or Collector.

Transfer

A value of Transfer means that any updates to the core store data, or any stock transfers or requisitions into that store, are synced to the corresponding store instance on the selected sync site. No other transactions or other store-specific data will be transferred. The store instance on the selected sync site will normally be of type Transfer.

  • In a standard (non-mirrored) setup, the primary server is the sync server
  • In a mirrored setup, the primary server is like one of the remote sites, except that:
    • it can also control system data
    • like the sync server, it has copies of all stores

An example of a store on the sync server which needs to also be reportable on another remote site.

Store A exists as an Active store on sync server and as a Collector store on remote site 1:

  • On sync server store-specific data for this store can be entered. It can also receive stock transfers or requisitions from other stores active on the same sync site (sync server). This data is then synced to remote site 1.
  • On remote site 1 store-specific data for this store cannot be entered. However, the store can receive stock transfers or requisitions from other stores active on the same sync site (remote site 1). These will be synced to sync server.

An example of a store on one remote site which needs to receive stock transfers from a store on another remote site.

Store B exists as a Collector store on sync server, an Active store on remote site 1, and a Transfer store on remote site 2:

  • On remote site 1, store-specific data for this store can be entered. It can also receive stock transfers or requisitions from other stores active on the same sync site (remote site 1). This data is then synced to sync server.
  • On sync server, store-specific data for this store cannot be entered. However, it can receive stock transfers or requisitions from other stores active on the same sync site (sync server). These are then synced to remote site 1.
  • On remote site 2, store-specific data for this store cannot be entered, and is not synced from sync server. However, it can receive stock transfers or requisitions from other stores active on the same sync site (remote site 2). These are then synced to sync server, and then to remote site 1.

Another example of a store on one remote site which needs to receive stock transfers from a store on another remote site.

Store C exists as a Collector store on sync server, an Active store on remote site 2, and a Transfer store on remote site 1:

  • On remote site 2, store-specific data for this store can be entered. It can also receive stock transfers or requisitions from other stores active on the same sync site (remote site 2). This data is then synced to sync server.
  • On sync server, store-specific data for this store cannot be entered. However, it can receive stock transfers or requisitions from other stores active on the same sync site (sync server). These are then synced to remote site 2.
  • On remote site 1, store-specific data for this store cannot be entered. However, it can receive stock transfers or requisitions from other stores active on the same sync site (remote site 1). These are then synced to sync server, and then to remote site 2.

An example of a store on a remote site which needs to also be reportable on the sync server.

Store D exists as an Active store on remote site 2 and as a Collector store on sync server:

  • On remote site 2, store-specific data for this store can be entered. It can also receive stock transfers or requisitions from other stores active on the same sync site (remote site 2). This data is then synced to sync server.
  • On sync server, store-specific data for this store cannot be entered. However, it can receive stock transfers or requisitions from other stores active on the same sync site (sync server). These are then synced to remote site 2.

An example of a store which is local to the sync server only.

Store E exists only on sync server (and primary server, if different):

  • On sync server, store-specific data for this store can be entered. It can also receive stock transfers or requisitions from other stores active on the same sync site (sync server). This data is not synced anywhere.

An example of a store which is local to a single remote site.

Store F exists on remote site 1, but it must also exist on the sync server (and primary server, if different) as a Transfer store, so that it can be centrally managed:

  • On remote site 1, store-specific data for this store can be entered. It can also receive stock transfers or requisitions from other stores active on the same sync site (remote site 1). This data is not synced anywhere.

The table below defines which data types are considered as system or store-specific. Other types of data (e.g. preferences, user permissions are always local to the sync site, and are not synchronised).

Data Sync records Notes
Items System Including item-related data e.g. item categories, units
Names (except patients) System Including name-related data e.g. name categories, contacts
Visibility of items and names System
Merging of items and names System Except patients
Groups and departments System
Item master lists System
Budgets and accounts System
Transaction categories System
Purchase order categories System
Location types System
Customer stock history and requisitions Store
Locations Store
Stock and stocktakes Store
Transactions Store Including other transaction-related data e.g. inventory adjustments, backorders, builds
Purchase Orders Store Unless the Procurement is done centrally preference is switched on, in which case purchase orders can be entered on the primary for any store, regardless of its sync type
Goods received Store

These are a special case:

  • changes to the store record itself (all except the Synchronisation tab) are allowed only on the primary server, and then propagate on to any other sync sites as determined by the “syncs with” settings for that store on the sync server (i.e. to any other sync sites where “syncs with” is not None)
  • the store's sync preferences (the Synchronisation tab) are controlled on the sync server
  • a store should only ever be Active on one sync site at a time (usually on the site where it is Local)

If this preference is switched on in the primary site, then purchase orders can be entered for any store on the primary, regardless of its sync type (i.e. not just for active stores).

If it is switched on in a satellite, then purchase orders cannot be entered for any store on the satellite, even if it is active.

When this preference is changed on the primary, the change will propagate to all the satellites as well.

It is still possible to edit the preference on the satellite (e.g. to allow local procurement for stores active on that site), but if that is the case, then the related permissions for purchase orders for those stores on the primary will need to be disabled manually in order to prevent purchase orders for those stores being editable on both sites.

Prescriptions

By default, prescriptions and any other operations in dispensary mode which affect stock levels are not synced to the sync server, unless the Include transactions in sync option is enabled in the store sync preferences, as these can potentially generate a lot of sync traffic and there is usually no need for the central users to have this level of detail - the same site will usually have another store which supplies the dispensary and its transactions will be synced, and that's usually enough to infer usage for the dispensary store.

If this preference is switched off in a dispensary store on a remote site, the following data will not be synced back to the central sync server:

  • Transactions (including backorders, builds etc.)
  • Prescriptions
  • Stock and stocktakes

If the preference is switched on, all of the store's transaction & stock data will be synced to the sync server.

Patients

Patients in mSupply are a special kind of customer, but for the purposes of synchronisation, we need to treat them differently - more like store-specific data rather than system data. This is because patient data doesn't needs to be shared with all other sites.

Patients can be edited on any site having at least one Active dispensary store, and patient records will be synced to the primary server. Newly created patients will also be made visible in any dispensary stores on the primary server, depending on the store's visibility preferences. Subsequently, patient visibility is controlled from the primary server in the same way as other name (customer & supplier) records.

At the moment, patient records are not shared with other dispensary sites although this may change in future e.g. if there is a country-wide patient registration system, in which case we would have to assign a store to each patient in order to control where it can be edited. Otherwise, if the same patient record existed on more than one dispensary site, then either site could update it and that would update the copy on the primary server, but the copy on the other site would remain untouched.

Other dispensary data

Prescribers, abbreviations, patient medication records, and repeats are also treated as a special kind of store-specific data, similarly to patients i.e. they are editable on any site having an active dispensary store, and synced back to the primary server, but not shared with dispensary stores on other sites.

Transfers occur when there are two stores involved in a transaction, and includes stock transfers, requisitions (from a mobile store) and internal requisitions (from another desktop store). In a syncing system, very often these two stores are not active on the same site, so there has to be extra processing to ensure that both halves of the transaction are synced to both the initiating store/site and the responding store/site.

Both stores need to exist on both sites, and usually the initiating store is a transfer store on the responding site, and vice-versa. As of mSupply v3.83, unless both stores are active on the same site (in which case, everything can be done locally), the bulk of the processing is done on the central sync server when it detects the initiating half of a transfer transaction. In simple terms, the logic is something like this:

  1. when the sync server detects the initiating half of a transfer transaction
    1. it creates the responding half of the transaction, but with a dummy invoice/serial number of -1
    2. it ensures that both halves of the transaction are synced to both initiating and responding sites
  2. when the responding site receives the responding half of the transaction
    1. it assigns the next invoice/serial number for the store and sends that back to the sync server
    2. it also creates a log message for the initiating half of the transaction and sends that back to the initiating site via the sync server
  3. any subsequent changes to the initiating half of the transaction (usually very limited in scope) on the initiating store/site will be synced back to the sync server (according to the normal sync rules)
  4. any subsequent changes to the responding half of the transaction on the responding store/site will be synced back to the sync server (according to the normal sync rules)
  • This is where the customer in a customer invoice is another store
  • The initiating half of the transaction is where the customer invoice is finalised in the initiating store/site
  • The responding half of the transaction is a supplier invoice (on hold by default) in the responding store/site
  • This is where the supplier in a supplier (aka request) requisition is another store
  • The initiating half of the transaction is where the supplier requisition is finalised on mobile
  • The responding half of the transaction is a response requisition in the responding store/site
    • this shows up as a customer requisition on mobile if the responding store is active on another mobile site
    • or as a response requisition if the responding store is active on a desktop site
      • the user can create one or more customer invoices to fulfil the requistion
      • these customer invoices (when finalised) will generate corresponding supplier invoices back in the initiating store as stock transfers
  • This is where the supplier in a purchase order is another store, and is a two-stage process
  • The initiating half of the first transaction is where the purchase order is confirmed in the purchase order store/site (i.e. where the purchase order is editable)
    • if centralised procurement is enabled, this will be the primary site, otherwise it will be the normal initiating store/site (i.e. where the purchase order's store is active)
  • The responding half of the first transaction is a customer invoice in the responding store/site
  • The initiating half of the second transaction is where the customer invoice is finalised in the responding store/site
    • note that adding extra customer invoice lines before the invoice is finalised will create corresponding new goods received lines
  • The responding half of the second transaction is a goods received record in the initiating store/site (i.e. where the purchase order's store is active)
  • If centralised procurement is enabled:
    • the received quantities for the original purchase order lines will be updated on the primary whenever the corresponding goods received lines are received there (and forwarded from there to the initiating store/site)
    • when the primary receives any subsquent updates to goods received lines from the initiating store/site, it will update the quantities in the corresponding purchase order lines
  • If centralised procurement is not enabled, the received quantities for the original purchase order lines will be updated in the initiating store/site when it receives the goods received lines from the primary
  • On the sync server if you login in supervisor mode, you can then run reports on one or more stores. The reports can answer questions such as:
    • How much stock on hand of item X (or all items) are there at each location?
    • What is the value of stock on hand across the whole system?
    • How many of item X is being used each month at each location?
  • When the synchronisation system is turned on, a new Special > Users report… > Sync report… menu item is available. Choosing this will create a report showing the last time each of your sync stores connected to the central sync server. Those that connected more than a month ago will be highlighted in red.
  • A license from Sustainable Solutions to use the web server module and the synchronisation module on the primary server.
  • Internet at each sync site. This doesn't have to be on all the time for each satellite server, but at least an hour or so per day or per week (depending on the transaction volume and the speed and latency of the internet, and how often you need primary server data updated). Obviously, the internet needs to be on at the primary server at the same time.

If there are queued sync records, there will be a message at the bottom left of the navigator which shows the number of sync records in the queue. If this is not going down, there may be an entry in the log.

  • Choose Help > About mSupply
  • At the bottom left of the window there is a list of tables and the number of records in each table.
  • Scroll the list to near the bottom, and you will see the number of records in the sync_out table.
    • If the number is zero, your copy of mSupply is up to date
    • If the number is growing from day to day, there are possible reasons:
      • you need to provide more internet time or faster internet
      • there may be a problem that needs the attention of Sustainable Solutions. Before contacting us, make sure the internet is connected for an hour, and see if the number is decreasing or not.

This is not something for end users to configure, so it's best left to the experts at Sustainable Solutions! However, there is a lot of preparation ground work that can be done beforehand:

  1. Decide how to configure your server(s)
    • for a local combined primary and sync server:
      • can your local server be online most of the time (it is more important that the internet to be reliable than for it to be fast)?
      • have you a fixed external IP address?
      • can you open the necessary firewall ports to allow access to the local server from outside?
    • for a combined primary and sync server on the cloud:
      • is your local internet fast enough to support remote access to the cloud server to administer system data?
    • otherwise, it's better to have a separate local primary server and cloud sync server
  2. Do a site survey for all the proposed stores
    • how is the internet access and local network?
    • how reliable is the power supply?
    • is there a suitable location for mSupply?
  3. Decide which version of mSupply is best for each store - don't worry, this can be changed later and licenses can be transferred
    • central primary server: mSupply server with local clients (depending on the number of users)
    • central sync server: mSupply server with local clients (depending on the number of users) and web server
    • multi-user central/district warehouse/busy dispensary: mSupply server with local clients (depending on the number of users)
    • single-user store/quiet dispensary (desktop): mSupply single-user desktop
    • small store/clinic: mSupply mobile
  4. Decide on the heirarchy of the stores you want to include
    • which stores supply other stores?
    • can stores which have the same supplying store transfer to each other, or just with the supplying store?
    • for dispensary stores, do you need to see prescription details on the central server
  5. Decide on the users for each site and their roles/permissions



  • Last modified: 2019/11/13 09:24
  • by Adrian Boone