Setup ODK-X Sync Endpoint with Cloud Services

This tutorial will help you launch ODK-X Sync Endpoint on a virtual machine hosted on a cloud service provider. ODK-X Sync Endpoint communicates with your ODK-X Android applications in order to synchronize your data and application files.

There are 3 main options that we have documented to set up ODK-X Sync Endpoint

Step 0: Acquire a domain name or subdomain

Running the ODK-X Sync Endpoint in the cloud will require access to a publicly registered domain name to allow for a secure connection between Android devices and the Sync Endpoint. Domain names can be purchased via many providers. We have used Google Domains, Amazon Route 53, Azure App Services Domains, and Cloudflare Registrar successfully.

If you already own a domain, you may add a subdomain record for use with Sync Endpoint without purchasing a whole new domain. Before you go on, make sure you have a domain and know how to log into your domain management console to add a DNS record!

Note

Specific instructions for connecting ODK-X Sync Endpoint to your domain will vary based on your registrar and DNS provider.

Option 1: DigitalOcean console

If you'd like to set up an ODK-X server that's accessible from anywhere via the Internet, DigitalOcean provides a one-click configuration that's nicely geared with nearly all the tools you'll need to set up your new server. The only thing it doesn't do is register a domain name, which you will have to do in order to obtain a security certificate for your server. These instructions walk you through:

Setting up a DigitalOcean account

  1. If you haven’t already, create an account on DigitalOcean.

Setting up a Droplet

  1. First, click on the Create dropdown button at the top right of the screen. Then, click on Droplet to create a droplet cloud server.

../_images/create-droplet.png
  1. In the Distributions tab, on the Create Droplet screen; select 18.04 (LTS) x64 under the Ubuntu dropdown. Next, choose a plan and data center region based on your needs.

Note

Sync Endpoint requires more than 2GB of space to run, this means that plans below 4GB will not work.

../_images/do-distribution.png ../_images/do-plan.png
  1. Scroll down to the Select additional options, click on the User data checkbox, copy and paste the contents of the cloud_init_DO.yml file in the text area provided.

../_images/do-userdata.png ../_images/do-userdata2.png
  1. The next step is Authentication. There are two authentication types to select from; SSH Keys and Password. We highly recommend that you use an SSH key for authentication. Copy and paste your SSH key username, and the key itself.

Use the following resource to learn more about creating an SSH key.

../_images/do-authentication.png
  1. After the authentication is set up, you can choose to name the droplet; then scroll down and click the Create Droplet button. This might take a few minutes to set up.

Setting up a DNS Record

  1. On the resources tab of the main DigitalOcean page, click on the Droplet you created.

../_images/do-droplets.png
  1. Obtain the IP address of the droplet you created.

  2. Log into your account for your domain name registrar and DNS provider. See Acquiring a domain name for more information and a list of registrars and DNS providers.

  3. Add a dns 'A' record for the domain or subdomain you would like to use for the Sync Endpoint with your droplet's IP address.

Connecting to your Droplet

  1. On the resources tab of the main DigitalOcean page, click on the Droplet you created.

../_images/do-droplets.png
  1. Now, click on the Console link in the upper-right corner of the page

../_images/do-console.png
  1. A console window will now open up. If you chose the password authentication, you will be asked to enter your username and then asked for a password.

../_images/do-console-terminal.png
  1. Before running our launch scripts, we need to check our logs to ensure that all the packages have been successfully installed, which should take about 2-3 minutes. The droplet may also reboot in this time.

Use the following command to get into the log directory.
$ cd /var/log

Now, open the log file with command:

$ tail cloud-init-output.log

If you see the message “The system is finally up, after X seconds” you can proceed to the next step! Otherwise, continue to wait and check the log file again.

  1. In order to run our launch scripts, we must first navigate back to the root directory with the following command:

$ cd /root

Now, we can run our build scripts with the command:

$ ./script_to_run.sh

The script will ask you for the server's domain and an administration email address to configure https on the server.

After gathering this data the script will begin the install and you should see a bunch of statements executing in your console. Wait approximately 5-10 minutes for the installation to complete.

../_images/do5.png

Once all the services have been created, we need to check if all the services are running properly with the command:

$ docker stack ls

If there are 8 (or 7 without https) services running under the name syncldap, everything is running properly.

  1. From the Droplets section of the console, obtain the IP address of the droplet you created. Now, navigate to https://[IP_ADDRESS]:40000 within your browser in order to access the services screen. It will warn you about your connection not being private but should give you the option to proceed at the bottom.

../_images/do6.png ../_images/do7.png
  1. If you see the following screen after proceeding, you are good to go!

../_images/do8.png
  1. Read our section on Creating a Sample User to learn how to create a user from within the admin interface. This section can be found here.

Enabling a firewall to prevent unintended traffic

  1. On the DigitalOcean console, click on the arrow beside the MANAGE dropdown and navigate to the Networking section. Go to the Firewalls section and click Create Firewall.

../_images/do-networking.png
  1. Set a name for your firewall and modify the inbound rules to match the inbound rules specified in the picture below (SSH, HTTP, HTTPS and port for admin interface). Attach the firewall to the desired droplet. Leave the outbound rules as-is.

../_images/do10.png ../_images/do11.png
  1. After going through the instructions for “Creating a Sample User,” we no longer need access to this admin interface anymore. This admin interface is running on port 40000, and in order to ensure that this admin interface is not publicly accessible to anyone, we want to remove the rule that accepts incoming traffic to that port. Go ahead and remove the following rule:

../_images/do12.png

Launching the ODK-X Server

  1. Navigate to http://[IP_ADDRESS]/web-ui/login in order to access the login screen.

../_images/do13.png

Once a user has been created in the admin interface, this is the login screen that the user will use to log in and access their data.

Option 2: Azure console

We have noticed that sync-endpoint runs the smoothest on Azure. These instructions will walk you through the following:

Setting up an Azure account

  1. If you haven’t already, create an account on Azure.

Setting up a virtual machine

  1. First, click on the Virtual Machines button underneath the Azure Services section on the portal. Then, click on Add to create a new virtual machine.

../_images/azure1.png ../_images/azure2.png
  1. Create a new resource group to attach to this virtual machine by clicking on Create new. Additionally, enter a name for the virtual machine and make sure that Ubuntu Server 18.04 LTS is selected for the image name.

../_images/azure3.png
  1. Scroll down and select your authentication type. We highly recommend that use an SSH key for authentication. Copy and paste your SSH key username, and the key itself.

Use the following resource to learn more about creating an SSH key.

../_images/azure4.png
  1. Click the Advanced tab at the top and copy and paste the contents from the cloud_init_AZURE.yml file into the Cloud init box. Finally, click Review + create to actually create the machine.

../_images/azure5.png
  1. In order to modify the firewall settings and change the type of incoming traffic we want to allow, we need to modify the Networking settings of our VM. Navigate to this section and then add an inbound security rule that matches the rule below. Leave the outbound rules as-is.

../_images/azure6.png

Setting up a DNS Record

  1. Within the Virtual Machine overview section, locate the IP address of your machine.

../_images/azure7.png
  1. Log into your account for your domain name registrar and DNS provider. See Acquiring a domain name for more information and a list of registrars and DNS providers.

  2. Add a dns 'A' record for the domain or subdomain you would like to use for the Sync Endpoint with your droplet's IP address.

Connecting to your virtual machine

  1. Within the Virtual Machine overview section, locate the IP address of your machine.

../_images/azure7.png
  1. Open up a terminal window and enter the command

$ ssh -i PATH_TO_PRIVATE_KEY USERNAME@IP_ADDRESS

The first parameter represents the path to your private key you used for SSH authentication, the second parameter the username you used for SSH authentication, and the final parameter the IP address of the virtual machine.

  1. Before running our launch scripts, we need to check our logs to ensure that all the packages have been successfully installed, which should take about 2-3 minutes. The virtual machine may also reboot in this time.

Use the following command to get into the log directory.
$ cd /var/log

Now, open the log file with command:

$ tail cloud-init-output.log

If you see the message “The system is finally up, after X seconds” you can proceed to the next step! Otherwise, continue to wait and check the log again.

  1. In order to run our launch scripts, we must first navigate back to the home directory with the following command:

$ cd /home

Now, we can run our build scripts with the command:

$ sudo ./script_to_run.sh

The script will ask you for the server's domain and an administration email address to configure https on the server.

After gathering this data the script will begin the install and you should see a bunch of statements executing in your console. Wait approximately 5-10 minutes for the installation to complete.

../_images/azure8.png

Once all the services have been created, we need to check if all the services are running properly with the command:

$ docker stack ls

If there are 8 (or 7 without https) services running under the name syncldap, everything is running properly.

  1. After obtaining the IP address of the virtual machine you created, navigate to https://[IP_ADDRESS]:40000 within your browser in order to access the services screen. It will warn you about your connection not being private but should give you the option to proceed at the bottom.

../_images/azure9.png
  1. If you see the following screen after proceeding, you are good to go!

../_images/azure10.png
  1. Read our section on Creating a Sample User to learn how to create a user from within the admin interface. This section can be found here.


  1. After going through the instructions for Creating a Sample User, we no longer need access to this admin interface anymore. This admin interface is running on port 40000, and in order to ensure that this admin interface is not publicly accessible to anyone, we want to remove the rule that accepts incoming traffic to that port. We do this the same way we added the rules above.

Launching the ODK-X Server

  1. Navigate to http://[IP_ADDRESS]/web-ui/login in order to access the login screen.

../_images/azure11.png

Once a user has been created in the admin interface, this is the login screen that the user will use to log in and access their data.

Option 3: Amazon Web Services console

Setting up an AWS account

  1. If you haven’t already, create an account on Amazon Web Services.

Setting up a virtual machine

  1. First, click on EC2 link under the COMPUTE section. Then, go ahead and launch a new instance.

../_images/aws1.png ../_images/aws2.png
  1. You must start by choosing an Amazon Machine Image (AMI). Scroll through the options and select Ubuntu Server 18.04 LTS (HVM), SSD Volume Type which should be the fifth option from the top.

../_images/aws3.png
  1. Skip the “Choose an Instance Type” step. Instead, click on the 3: Configure Instance tab at the top and then attach the cloud_init_AWS.yml file we provided within the User data section under “Advanced Details.”


  1. Click on the 6. Configure Security Group tab in order to modify the firewall rules and control the traffic for the instance. Create a new security group and modify the rules to match the rules specified below, then click Review and Launch.

../_images/aws4.png
  1. Review the Instance Launch and then click Launch. Now, create a new key pair to access your instance via SSH and make sure to download it to a secure location. Finally, click Launch Instances!

../_images/aws5.png

Setting up a DNS Record

  1. From the EC2 dashboard and click on Running instances.

../_images/aws6.png
  1. Select the instance you just created, and obtain its public IP address.

  2. Log into your account for your domain name registrar and DNS provider. See Acquiring a domain name for more information and a list of registrars and DNS providers.

  3. Add a dns 'A' record for the domain or subdomain you would like to use for the Sync Endpoint with your droplet's IP address.

Connecting to your virtual machine

  1. Go back to the EC2 dashboard and click on Running instances.

../_images/aws6.png
  1. Select the instance that you want to connect to and then click Connect.

../_images/aws7.png
  1. Open up a terminal window and enter the following command to change key permissions.

$ chmod 400 KEY_NAME.pem

Now, use the following command in order to SSH into your virtual machine.

../_images/aws8.png
$ ssh -i “KEY_NAME.pem” PUBLIC_DNS
  1. Before running our launch scripts, we need to check our logs to ensure that all the packages have been successfully installed, which should take about 2-3 minutes. The virtual machine may also reboot in this time.

Use the following command to get into the log directory.
$ cd /var/log

Now, open the log file with command:

$ tail cloud-init-output.log

If you see the message “The system is finally up, after X seconds” you can proceed to the next step! Otherwise, continue to wait and check the log again.

  1. In order to run our launch scripts, we must first navigate back to the Ubuntu directory with the following command:

$ cd /home/ubuntu

Now, we can run our build scripts with the command:

$ sudo ./script_to_run.sh

The script will ask you for the server's domain and an administration email address to configure https on the server.

After gathering this data the script will begin the install and you should see a bunch of statements executing in your console. Wait approximately 5-10 minutes for the installation to complete.

../_images/aws9.png

Once all the services have been created, we need to check if all the services are running properly with the command:

$ docker stack ls

If there are 8 (or 7 without https) services running under the name syncldap, everything is running properly.

  1. After obtaining the IP address of the virtual machine you created, navigate to https://[IP_ADDRESS]:40000 within your browser in order to access the services screen. It will warn you about your connection not being private but should give you the option to proceed at the bottom.

../_images/aws10.png
  1. If you see the following screen after proceeding, you are good to go!

../_images/aws11.png
  1. Read our section on Creating a Sample User to learn how to create a user from within the admin interface. This section can be found here.


  1. After going through the instructions for Creating a Sample User, we no longer need access to this admin interface anymore. This admin interface is running on port 40000, and in order to ensure that this admin interface is not publicly accessible to anyone, we want to remove the rule that accepts incoming traffic to that port. We do this the same way we added the rules above.

Launching the ODK-X Server

  1. Navigate to http://[IP_ADDRESS]/web-ui/login in order to access the login screen.

../_images/azure11.png

Note

If you are unable to log in, you may need to take the docker stack down and bring it back up again. That can be done with the following commands below:

$ docker stack rm syncldap
$ docker stack deploy -c /root/sync-endpoint-default-setup/docker-compose.yml syncldap

Anonymous Access for ODK-X Sync Endpoint Cloud

To Enable or Disable Anonymous User Access for your ODK-X Sync Endpoint follow these instructions.