Kwil
Search…
⌃K

Setup

While Kwil hopes that you let us manage your databases for you, we understand that this might not fit the requirements of all users! Therefore, we have made Kwil DB's open-source and free to use!

Requirements

Kwil Databases are responsible for handling a lot of separate processes. Therefore, in order to run your own, there will be a variety of requirements that are atypical for a centralized database.

Machine Requirements:

  • At least 2 vCPU
  • 4 GB Memory
  • Ubuntu 18.04 or later installed
  • (Recommended) 100 GB Storage
  • Any CLI Text Editor (We will be using Nano)

Others:

  • An Arweave private key holding some AR (JWK format).
  • A TLS/SSL certificate (Preferably Wildcard)
  • A Domain Name

Getting started

Once you have all of the required resources, you're ready to set up your node! Open up your terminal to the directory where you want your node files to be stored. We will begin by updating our system:
sudo apt-get update;
Next, we need to install both NGINX and Postgresql:
sudo apt-get install nginx -y;
sudo apt -y install postgresql postgresql-client postgresql-contrib;
Then, we need to install Node Version Manager:
sudo curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash;
Next, we need to export Node Version Manager:
export NVM_DIR="$HOME/.nvm";
[ -s "$NVM_DIR/nvm.sh" ] && \. "$NVM_DIR/nvm.sh";
[ -s "$NVM_DIR/bash_completion" ] && \. "$NVM_DIR/bash_completion";
Now, we need to install NodeJS:
nvm install stable;
Now, we will need to install software to enable our server to run in the background. We will be using an NPM package known as "forever".
npm i -g forever
Finally, we will need to download the synchronizer:
git clone https://github.com/kwilteam/kwil_db_v2.git

Configuring TLS and CORS With NGINX

We will now set up our TLS Certificate with NGINX, as well as create our CORS configuration. This will involve us creating four new files, as well as editing a fifth:

Private Key

First, we will need to create a file to hold the actual private key, stored in PEM format:
sudo touch /etc/nginx/sites-available/privkey.pem;
Let's open up the file and paste in our TLS Certificate:
sudo nano /etc/nginx/sites-available/privkey.pem
In the editor:
-----BEGIN PRIVATE KEY-----
...
-----END PRIVATE KEY-----
Exit the file using "^x", and then press "y" to save, and then "return"

Full Chain

Next, we will need a file to hold the full chain PEM:
sudo touch /etc/nginx/sites-available/fullchain.pem;
Opening up the file to paste the full chain:
sudo nano /etc/nginx/sites-available/fullchain.pem;
In the editor:
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----

SSL NGINX Config

Then, we will create a file to hold our SSL NGINX configurations:
sudo touch /etc/nginx/sites-available/options-ssl-nginx.conf;
Then, open the file:
sudo nano /etc/nginx/sites-available/options-ssl-nginx.conf;
Then, paste in the following configuration:
ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
client_max_body_size 10M;
ssl_session_tickets off;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;
ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA><HE-RSA-AES256-GCM-SHA384";

Diffie-Hellman Params

Finally, we will create a file to specify how NGINX handles Diffie-Hellman key exchange. This should have been given to you by your SSL CA with the rest of your key:
sudo touch /etc/nginx/sites-available/ssl-dhparams.pem
Open the file:
sudo nano /etc/nginx/sites-available/ssl-dhparams.pem
Then, paste in:
-----BEGIN DH PARAMETERS-----
...
-----END DH PARAMETERS-----

Editing NGINX Configuration:

Once you have set up your TLS/SSL Certificate, you will need to update your NGINX configuration. This example will use a wide-open CORS policy with restricted filesystem access (with the exception of static assets meant to be served):
sudo nano /etc/nginx/sites-available/default
Paste in the following. Make sure to change the server_name as well as the user for the /public location:
server {
root /var/www/html;
index index.html index.htm index.nginx-debian.html;
server_name example.com www.example.com;
#If you have a wildcard SSL certificate, change server_name to *.example.com
location /public {
autoindex on;
alias /home/{user}/Kwil_DB_Synchronizer/public;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
add_header Access-Control-Allow-Origin *;
}
location / {
proxy_pass http://localhost:1984; #whatever port your app runs on
proxy_http_version 1.1;
add_header 'Access-Control-Allow-Credentials' 'true';
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';
add_header Access-Control-Allow-Origin *;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_cache_bypass $http_upgrade;
}
listen 80;
listen [::]:443 ssl ipv6only=on;
listen 443 ssl;
ssl_certificate /etc/nginx/sites-available/fullchain.pem;
ssl_certificate_key /etc/nginx/sites-available/privkey.pem;
include /etc/nginx/sites-available/options-ssl-nginx.conf;
ssl_dhparam /etc/nginx/sites-available/ssl-dhparams.pem;
}
Exit by pressing "^x", "y", then "return".
Now, we're done setting up our NGINX configuration! We just need to restart the server:
sudo service nginx restart;
If this fails, run this command to debug:
sudo nginx -t -c /etc/nginx/nginx.conf;
Lastly, you will need to point your domain name to the external IP address of the machine you are running on. To do this, all you need to do is add an A record with your name service provider!

Setting Up The Database

Now that we have set up NGINX, we need to set up Postgresql. Since we installed Postgresql earlier, all we need to do is change our password and restart!
sudo -u postgres psql postgres;
In Postgresql:
\password postgres
(Enter new password)
(Re-enter password)
\q
Now, just restart Postgresql:
sudo service postgresql restart;

Setting Up The Synchronizer

The last thing we need to do is set up the Kwil DB Synchronizer. To begin, lets enter the folder that we downloaded it to:
cd kwil_db_v2
Install necessary NPM packages:
npm i;
Now, it's time to configure the synchronizer. Start by just running the server:
npm start
This should throw an error. Our team recently found a config issue (April 10th), and so as a quick workaround, the first failure should fix the issues.
Once the server has failed to start once, you need to fill out your .env configuration. The application should provide a base template.

.env Configuration

Below is an example of what the base template fully filled out would like:
ARWEAVE_GRAPH_HOST = https://arweave.net
NODE_PORT = 1984
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_USER=postgres
DATABASE_PASSWORD=password
DATABASE_NAME=postgres
DATABASE_CONNECTOR= postgresql
NODE_ENV = production
BUNDLE_VERSION = 1.3
SYNCED_DATA_MOATS = test
SYNC = false
SHOVE = false
ALLOW_REGISTRATION = false
UPCHARGE_RATE = 1.3
KEY_SECRET = mysupersecretcipherkey
PRIVATE_KEY = 2056d52c3fa968e426a3424a21f8020092e5075bcad423805fbd382bcff20a65
REGISTRY_ADDRESS = https://registry.kwil.xyz
ACCEPTED_TOKENS = USDC_ethereum USDC_polygon KRED_polygon

Arweave JWK

Next, you need to paste in an Arweave JWK to key.json:
{"kty":"RSA","n":"ucro-D1CW6FZlMqjB_Ezz_1SnCzRqSFRzn6H9XEWzYT-Wo4Uu8zme2RR1vNaaJ7cA2eXBUPISWo9Us8BY0_kBXFd67W8o4vzNfJAm-SL4jrq8Sgl1aI0iMMplREX8KEus94iYVJFd2_OaExRdrJ3SdIgEY5MwBFjZdJ7bPPqK_TC-xtzNiwyC9zd1hIOE2txISl-g-jtZyOHt3EsZMTU92S-aLXg23Vd7NLyURSjH2vvfbrzabnfVTbZHQsiD1U_c-HVenGPEkFUQLTRyXKihRfUe0lHmHRO7rZ6Wb4KTOFs2VjXM4hSgjUAxhuAz4FRUUdEITgbDsEdyo60JnwBGwWcQw5qzz4cQxBRixpvggvQBZEbHYCFEza0bh8tFRjQtR-sBPI_P6oPdIbtC78IHDllg5J0Q8SHUDD9bupMjw74k0HWounyTiTHEGgLDVYSd67FURt947PRuz-jVqL72s5FpNd4NBa7QMRaajiLp0dX57wy8yjHutTWUODdgd9N24d8Bt0NUTN_LlHY2VaAngzZITWkaEGjHIPXkTRUdlhFf8JRYsO5u-CR5cPHp6Hwh7WviSpqGzDNBasph13vGfYx0USe1IUMDD2EyIgBpixuiqZ6d-a8bYvW4hkLqamO-cENc7sBZshVD0VQ0onBpShqKI136FSAWXFuo0beQHk","e":"AQAB","d":"P7ZPkeEgAmAgYh_jLHQQeoC6dYgBFeixLwjBAvAXC79RoigbABPR7fqG_hlRpZLki95YMZv-8P4XrXADX9UjsqIfvRcDa2owqw06DrufJKNjsfM8ljgbn9KdTdsgxInhhBjm3UzjOYQD2kxhmzy0H4KJMLwFoc_UsDQv0zsJ0K0MuCO_rJIXT24cPK7QWOR9VQvvsLiNwNK0VbnfqWa-SWAo06ubQmbLIM1t_1P-ZGYVKJ6Vnmn5JjW62nTY5z0ql0iYdna-w1nX1KCtWFGRXebAHM8SWDt6Xe4BowjCa37aXuZSVf-nm7MlYspHr-D-y8qOmH2u-a5L0wyAxrm7fryF_poAgy3wzg1d48tLCyqlNYCt_hicUkklHki7WpMs2ftQ5On6gXqjm3Vvyb0QUuEfhc6lITxw6lT9_vOnLw7SE_2lDF_guIQix8F0FgZb0VjMmGoLwhGjg4ylHz4zgySl6ytlUnBUo69_MHZBm7tXADsi2GZxI-J8EAIwg8q1HPisQwGdfAhjOge0kGf4F1pnAUUftL_auJu6c3C0azDYsOm_Slp-UdAvQEIWLYr8XNdLmu2MFcXmqAWZqTIfd158C3oHFhWNa5-QeP3ekT_ZmC5tyAk-V21-ZT_EQYFRdZKSEvsaW26fL-dIUpLwE4yanRyIwI_Ju5ppay71e2k","p":"7xnn9Yhxk4DsjtCf2JU6-wIyFOF7GSQ4XRuUrza27mCO8XJd6UmBkjsGibjHBkDetVNqW0qz1QbGRJ8Sj5GFZfFf8MkwzwGNlsUhOFXHou01EDIHyHXdXA_vgAoUYVujOIRqocDt_YeoS3hwopt-0W4YGJxX7mJnw61TyK7hNiVLu0m3HL0vWSdQBQtBeNkkxhfJd7u0YkhKNPJKnTPaYVmKffLebOMFhOtW-h8RI_2AefrgAI-yus6J5NfAAXgLPh21YCJ6I32MvhcET0B7wg1iJ2Yr39C9k5ilk7CLQ_ratjrxvobHLwLcZPrNPfdxS1KPDMxOWY6XJ6wNrSK_Yw","q":"xux724rwN5TMskHp8fhGrpBzNdW6HOZNCfJvKV2ODBlh738oShmVqjyQg7eMCsrXKcgRzziCLiC1mORdSt91NqZvwpXosnkY5KHdRvKJ51czk4ywSoggQK8Iai6G5-b2W87xs_HZEX0uTpzDmVTx-nMw0zCvfUQPKJSayat49ff4u8G6JN3cwcUgIQhHMGpEPAsuu6R46wCZmVWw5nSbcuzCGA5egD-ekWdbFJVLuzvRfqJAIDpILdybk0TUX0J7kkpDDcKWm-3w04-o21TBumE_u9DyzS_jgRK27RZk44dUmiWqrx6IcP13iRoDzQYosVlfyHUj6xuMP3N3rwDNcw","dp":"elJOkRaUxFmYrBefLzZpTjD5XiVczKfzDe8mSd08CQftGSjhUpAAcccwuRSG_XQCe9GI_9tf1QP6OvEGldxk2pp58r168uxge1mufRulXnzsQLTXtn5WGNsuM9r0QjqXbjqQMuB44NOeDb44IXhbkNk6Cp0xfwto-nqZ-nk9Tm4WKGK2A0XmCz5JecsgW6r7XTyeHVsyFXXWQPATrAjzGAqvB3BQynP6Rt3_SUbLhhYSvFTUdHUPgvJXKXQXDqlTVxvrxprc0XtRnY447jKXHdeq920V1RL9aZ0WWFBhbw3PBptxQX_CNxLHngGRLrqAg4BhHrMHAlooimtDM-gq8Q","dq":"fFHDPnvgkPv_ydAD42Nk2SgRuK1v-Su9LeBDtLhxVAJxAHqZtWWz8YaUUp1dekk2mzgcWUNZqHZsx7bAz1BEMvllk1iY6yFfe1PltLdTlC8dXrNoYcwdTA-X1LRrP3dKtCnrSKh3HsT7McorPCKob_uZogMlScZFKvlkm3D80cw9uSnjqbK2iqsl3z8LLqjbKCNc-4xYvZXMaNXzDrQFipeI3fvs933LHdhYRc7gW42Q6fGA3iNOhMsAt9MmXrpv_AdJTMo10hGslquLvjeEM3UfncOCWtX4-uu1Qwrh1oj3SLcNqxAL-YGPOTH2-iVFPUE76U28Ou0mg2kofUilHQ","qi":"LOyvpaLIyfD3KwdKz38kzuFc0RRQPwwa1qMUS-K4-GBclrdn7sSO6I8k2qTSMtpfPgsDWxf8jlx3oakma3J7lSh9Qcs7CUWpo_w4MqJwPfwVKnjkyBSQTqRU6PzNZALVEwO4aFfeko_JqeqMclMzfq5iRLrSSS56bGfYBpFzbaYg5EYtP-KLnOZCY5gJznHET9MMSIEuc39cPpwkkqwq4JQ_cfozRm9AjWKRur98ZLqFQRfM6mbAjc0w13jV1PgujLG3IeNAiNYFnrWjR-M68nmg01rQwdHnRZl7tUo7tMZ2VXrxHOCk5BNtsQytyT8yqsKzeNM6_Ht-Vpe7QvZuYg"}
Now, we're ready to test to see if everything worked! Let's start the server:
npm start
The server will then go through syncing with the network. If your network has lots of data, this can take up to an hour. If your network is new, it should only take a few seconds.
Tip: If you're curious to just see if you set it up correctly, you can set "SYNC" and "SHOVE" to "false" in .env. This will prevent network synchronization and is useful for testing / saving on gas costs.
If everything worked correctly, you should see "Synchronizer is running on port 1984". If so, then all you need to do is stop the server and re-run with "forever":
forever start server.js;
And now you're done! Once the server finishes synchronizing, you should be able to access it via https with the API by using your domain name!