Setup ODK-X Sync Endpoint with Cloud Services

This tutorial will help you launch ODK-X Sync Endpoint on a virtual machine hosted on a cloud service provider. ODK-X Sync Endpoint communicates with your ODK-X Android applications in order to synchronize your data and application files.

There are 3 main options that we have documented to set up ODK-X Sync Endpoint
Easiest (recommended, especially for Windows machines)
More customization, setting up Virtual Machine using:

Step 0: Acquire a domain name or subdomain

Running the ODK-X Sync Endpoint in the cloud will require access to a publicly registered domain name to allow for a secure connection between Android devices and the Sync Endpoint. Domain names can be purchased via many providers. We have used Google Domains, Amazon Route 53, Azure App Services Domains, and Cloudflare Registrar successfully.

If you already own a domain, you may add a subdomain record for use with Sync Endpoint without purchasing a whole new domain. Before you go on, make sure you have a domain and know how to log into your domain management console to add a DNS record!

Note

Specific instructions for connecting ODK-X Sync Endpoint to your domain will vary based on your registrar and DNS provider.

Option 1: Using Python script to automatically set up a virtual machine on DigitalOcean

If you'd like to set up an ODK-X server that's accessible from anywhere via the Internet, DigitalOcean provides a one-click configuration that's nicely geared with nearly all the tools you'll need to set up your new server. The only thing it doesn't do is register a domain name, which you will have to do in order to obtain a security certificate for your server. These instructions walk you through:

Setting up a DigitalOcean account

  1. If you haven’t already, create an account on DigitalOcean.

Setting up a Droplet

  1. Use the following link in order to install the latest version of Python 3.0. Ensure that you are specifically installing an iteration of Python 3.0, as Python 2.0 will soon be deprecated. The installer should take about a minute to run.

Note

If using Windows, make sure to download the Windows version of Python instead.

  1. Open a terminal or command line. Install module to manage DigitalOcean droplets, using command:

$ pip3 install -U python-digitalocean

Note

Windows users also have the option of either using PuTTY, a free SSH client, in order to install the DigitalOcean module with pip. In the case that pip is not installed, Windows users can instead refer to the installation instructions from the following link and run the setup.py file to install the module instead.

  1. Generate API token by logging into DigitalOcean console and clicking on API under the MANAGE section. Now, click on Generate New Token and enter a name.

../_images/do1.png
  1. Download the following pyscript_DO.py and cloud_init_DO.yml files we have provided and ensure that they are located in the same directory. Switch to that directory and run the following command in order to set up your droplet:

$ python3 pyscript_DO.py [TOKEN] [NAME] [LOCATION]
[TOKEN] represents the token we obtained from step 3.
[NAME] represents the name that we want to give to our droplet.
[LOCATION] represents the desired data center location, and those codes can be found here.

Setting up a DNS Record

  1. From the DigitalOcean console, click on Droplets under the MANAGE section.

../_images/do2.png
  1. Obtain the IP address of the droplet you created.

  2. Log into your account for your domain name registrar and DNS provider. See Acquiring a domain name for more information and a list of registrars and DNS providers.

  3. Add a dns 'A' record for the domain or subdomain you would like to use for the Sync Endpoint with your droplet's IP address.

Connecting to your Droplet

  1. From the DigitalOcean console, click on Droplets under the MANAGE section.

../_images/do2.png
  1. Now, select your droplet and click on the Console link in the upper-right.

../_images/do3.png
  1. A console window will now open up. Enter your username and then you will be asked for a password. These credentials will be sent to the email associated with your DigitalOcean account. You will also be required to change the root password once you log in.

Note

Occasionally, Control + V may not work to paste the password, so you may have to right click and select paste.

../_images/do4.png
  1. Before running our launch scripts, we need to check our logs to ensure that all the packages have been successfully installed, which should take about 2-3 minutes. The droplet may also reboot in this time.

Use the following command to get into the log directory.
$ cd /var/log

Now, open the log file with command:

$ tail cloud-init-output.log

If you see the message “The system is finally up, after X seconds” you can proceed to the next step! Otherwise, continue to wait and check the log file again.

  1. In order to run our launch scripts, we must first navigate back to the root directory with the following command:

$ cd /root

Now, we can run our build scripts with the command:

$ ./script_to_run.sh

The script will ask you for the server's domain and an administration email address to configure https on the server.

After gathering this data the script will begin the install and you should see a bunch of statements executing in your console. Wait approximately 5-10 minutes for the installation to complete.

../_images/do5.png

Once all the services have been created, we need to check if all the services are running properly with the command:

$ docker stack ls

If there are 8 (or 7 without https) services running under the name syncldap, everything is running properly.

  1. From the Droplets section of the console, obtain the IP address of the droplet you created. Now, navigate to https://[IP_ADDRESS]:40000 within your browser in order to access the services screen. It will warn you about your connection not being private but should give you the option to proceed at the bottom.

../_images/do6.png ../_images/do7.png
  1. If you see the following screen after proceeding, you are good to go!

../_images/do8.png
  1. Read our section on Creating a Sample User to learn how to create a user from within the admin interface. This section can be found here.

Enabling a firewall to prevent unintended traffic

  1. On the DigitalOcean console, navigate to the Networking section under MANAGE Go to the Firewalls section and click Create Firewall.

../_images/do9.png
  1. Set a name for your firewall and modify the inbound rules to match the inbound rules specified in the picture below (SSH, HTTP, HTTPS and port for admin interface). Attach the firewall to the desired droplet. Leave the outbound rules as-is.

../_images/do10.png ../_images/do11.png
  1. After going through the instructions for “Creating a Sample User,” we no longer need access to this admin interface anymore. This admin interface is running on port 40000, and in order to ensure that this admin interface is not publicly accessible to anyone, we want to remove the rule that accepts incoming traffic to that port. Go ahead and remove the following rule:

../_images/do12.png

Launching the ODK-X Server

  1. Navigate to http://[IP_ADDRESS]/web-ui/login in order to access the login screen.

../_images/do13.png

Once a user has been created in the admin interface, this is the login screen that the user will use to log in and access their data.

Option 2: Azure console

We have noticed that sync-endpoint runs the smoothest on Azure. These instructions will walk you through the following:

Setting up an Azure account

  1. If you haven’t already, create an account on Azure.

Setting up a virtual machine

  1. First, click on the Virtual Machines button underneath the Azure Services section on the portal. Then, click on Add to create a new virtual machine.

../_images/azure1.png ../_images/azure2.png
  1. Create a new resource group to attach to this virtual machine by clicking on Create new. Additionally, enter a name for the virtual machine and make sure that Ubuntu Server 18.04 LTS is selected for the image name.

../_images/azure3.png
  1. Scroll down and select your authentication type. We highly recommend that use an SSH key for authentication. Copy and paste your SSH key username, and the key itself.

Use the following resource to learn more about creating an SSH key.

../_images/azure4.png
  1. Click the Advanced tab at the top and copy and paste the contents from the cloud_init_AZURE.yml file into the Cloud init box. Finally, click Review + create to actually create the machine.

../_images/azure5.png
  1. In order to modify the firewall settings and change the type of incoming traffic we want to allow, we need to modify the Networking settings of our VM. Navigate to this section and then add an inbound security rule that matches the rule below. Leave the outbound rules as-is.

../_images/azure6.png

Setting up a DNS Record

  1. Within the Virtual Machine overview section, locate the IP address of your machine.

../_images/azure7.png
  1. Log into your account for your domain name registrar and DNS provider. See Acquiring a domain name for more information and a list of registrars and DNS providers.

  2. Add a dns 'A' record for the domain or subdomain you would like to use for the Sync Endpoint with your droplet's IP address.

Connecting to your virtual machine

  1. Within the Virtual Machine overview section, locate the IP address of your machine.

../_images/azure7.png
  1. Open up a terminal window and enter the command

$ ssh -i PATH_TO_PRIVATE_KEY USERNAME@IP_ADDRESS

The first parameter represents the path to your private key you used for SSH authentication, the second parameter the username you used for SSH authentication, and the final parameter the IP address of the virtual machine.

  1. Before running our launch scripts, we need to check our logs to ensure that all the packages have been successfully installed, which should take about 2-3 minutes. The virtual machine may also reboot in this time.

Use the following command to get into the log directory.
$ cd /var/log

Now, open the log file with command:

$ tail cloud-init-output.log

If you see the message “The system is finally up, after X seconds” you can proceed to the next step! Otherwise, continue to wait and check the log again.

  1. In order to run our launch scripts, we must first navigate back to the home directory with the following command:

$ cd /home

Now, we can run our build scripts with the command:

$ sudo ./script_to_run.sh

The script will ask you for the server's domain and an administration email address to configure https on the server.

After gathering this data the script will begin the install and you should see a bunch of statements executing in your console. Wait approximately 5-10 minutes for the installation to complete.

../_images/azure8.png

Once all the services have been created, we need to check if all the services are running properly with the command:

$ docker stack ls

If there are 8 (or 7 without https) services running under the name syncldap, everything is running properly.

  1. After obtaining the IP address of the virtual machine you created, navigate to https://[IP_ADDRESS]:40000 within your browser in order to access the services screen. It will warn you about your connection not being private but should give you the option to proceed at the bottom.

../_images/azure9.png
  1. If you see the following screen after proceeding, you are good to go!

../_images/azure10.png
  1. Read our section on Creating a Sample User to learn how to create a user from within the admin interface. This section can be found here.


  1. After going through the instructions for Creating a Sample User, we no longer need access to this admin interface anymore. This admin interface is running on port 40000, and in order to ensure that this admin interface is not publicly accessible to anyone, we want to remove the rule that accepts incoming traffic to that port. We do this the same way we added the rules above.

Launching the ODK-X Server

  1. Navigate to http://[IP_ADDRESS]/web-ui/login in order to access the login screen.

../_images/azure11.png

Once a user has been created in the admin interface, this is the login screen that the user will use to log in and access their data.

Option 3: Amazon Web Services console

Setting up an AWS account

  1. If you haven’t already, create an account on Amazon Web Services.

Setting up a virtual machine

  1. First, click on EC2 link under the COMPUTE section. Then, go ahead and launch a new instance.

../_images/aws1.png ../_images/aws2.png
  1. You must start by choosing an Amazon Machine Image (AMI). Scroll through the options and select Ubuntu Server 18.04 LTS (HVM), SSD Volume Type which should be the fifth option from the top.

../_images/aws3.png
  1. Skip the “Choose an Instance Type” step. Instead, click on the 3: Configure Instance tab at the top and then attach the cloud_init_AWS.yml file we provided within the User data section under “Advanced Details.”


  1. Click on the 6. Configure Security Group tab in order to modify the firewall rules and control the traffic for the instance. Create a new security group and modify the rules to match the rules specified below, then click Review and Launch.

../_images/aws4.png
  1. Review the Instance Launch and then click Launch. Now, create a new key pair to access your instance via SSH and make sure to download it to a secure location. Finally, click Launch Instances!

../_images/aws5.png

Setting up a DNS Record

  1. From the EC2 dashboard and click on Running instances.

../_images/aws6.png
  1. Select the instance you just created, and obtain its public IP address.

  2. Log into your account for your domain name registrar and DNS provider. See Acquiring a domain name for more information and a list of registrars and DNS providers.

  3. Add a dns 'A' record for the domain or subdomain you would like to use for the Sync Endpoint with your droplet's IP address.

Connecting to your virtual machine

  1. Go back to the EC2 dashboard and click on Running instances.

../_images/aws6.png
  1. Select the instance that you want to connect to and then click Connect.

../_images/aws7.png
  1. Open up a terminal window and enter the following command to change key permissions.

$ chmod 400 KEY_NAME.pem

Now, use the following command in order to SSH into your virtual machine.

../_images/aws8.png
$ ssh -i “KEY_NAME.pem” PUBLIC_DNS
  1. Before running our launch scripts, we need to check our logs to ensure that all the packages have been successfully installed, which should take about 2-3 minutes. The virtual machine may also reboot in this time.

Use the following command to get into the log directory.
$ cd /var/log

Now, open the log file with command:

$ tail cloud-init-output.log

If you see the message “The system is finally up, after X seconds” you can proceed to the next step! Otherwise, continue to wait and check the log again.

  1. In order to run our launch scripts, we must first navigate back to the Ubuntu directory with the following command:

$ cd /home/ubuntu

Now, we can run our build scripts with the command:

$ sudo ./script_to_run.sh

The script will ask you for the server's domain and an administration email address to configure https on the server.

After gathering this data the script will begin the install and you should see a bunch of statements executing in your console. Wait approximately 5-10 minutes for the installation to complete.

../_images/aws9.png

Once all the services have been created, we need to check if all the services are running properly with the command:

$ docker stack ls

If there are 8 (or 7 without https) services running under the name syncldap, everything is running properly.

  1. After obtaining the IP address of the virtual machine you created, navigate to https://[IP_ADDRESS]:40000 within your browser in order to access the services screen. It will warn you about your connection not being private but should give you the option to proceed at the bottom.

../_images/aws10.png
  1. If you see the following screen after proceeding, you are good to go!

../_images/aws11.png
  1. Read our section on Creating a Sample User to learn how to create a user from within the admin interface. This section can be found here.


  1. After going through the instructions for Creating a Sample User, we no longer need access to this admin interface anymore. This admin interface is running on port 40000, and in order to ensure that this admin interface is not publicly accessible to anyone, we want to remove the rule that accepts incoming traffic to that port. We do this the same way we added the rules above.

Launching the ODK-X Server

  1. Navigate to http://[IP_ADDRESS]/web-ui/login in order to access the login screen.

../_images/azure11.png

Once a user has been created in the admin interface, this is the login screen that the user will use to log in and access their data.

Creating a Sample User

1. Start by logging into the ldap-service. Copy the login below.
- login DN: cn=admin,dc=example,dc=org
- password: admin
../_images/setup-user1.png
  1. Click the + sign next to dc=example, dc=org to expand it. Within the unfolded menu, in the ou=people section, click on Create a child entry (new person).

../_images/setup-user2.png
  1. Then, select the Generic: User Account template.

../_images/setup-user3.png
  1. Fill out information for the new user and “create object.” Assign it to the default_prefix_synchronize_tables group. Will need to commit (confirm) that you want to create this entry on the next screen.

../_images/setup-user4.png

We have now created the user! We just need to add the user to the respective group from the group settings.

  1. Click the + sign next ou=groups to expand it. Within the unfolded menu, in the ou=default_prefix section, click on gidNumber=503, which is the group ID that corresponds to default_prefix_synchronize_tables. Groups correspond to the access permissions available to a certain user.

../_images/setup-user5.png
  1. Click on Add new attribute which should show a pull-down menu and then select memberUid. Enter the memberUid of the user you just created, and then update the object.

../_images/setup-user6.png ../_images/setup-user7.png
  1. Navigate to http://[IP_ADDRESS]/web-ui/login in order to access the login screen.

../_images/setup-user8.png

Note

If you are unable to log in, you may need to take the docker stack down and bring it back up again. That can be done with the following commands below:

$ docker stack rm syncldap
$ docker stack deploy -c /root/sync-endpoint-default-setup/docker-compose.yml syncldap