Survey Form Deployment and Data Pipeline

28 Mar 2023 - Sudip Shrestha

This project is about collecting data by doing survey. Simply, survey data on social media usage is collected. We will be using deploying a single page that contains the survey form to the public internet. We will collect all the data on the MongoDB database. We will create an API that helps to fetch all the survey data from the MongoDB database. You can see the deployed survey form in this link.
After deploying the survey form, we will create a data pipeline that consists of fetching the data from the database and cleaning the data for further use in visualization and analysis. We will be using Airflow for scheduling these two processes one after another. Airflow is an open-source platform for developing, scheduling, and monitoring different workloads. It provides a graphical user interface to manage our workloads.

Code Availability
Survey Form here
Data Pipeline here

Technology Used
HTML, CSS, Javascript
Nodejs(version 16.17.0), MongoDB

Steps for deploying and pipelining

Step 1: Create a Survey Form
Create a directory named your_directory_name. Create a simple survey form in index.html about social media usage by individuals inside your_directory_name. The field includes the name, age, sex, social media preferred, and frequency of usage.

Step 2: Create a node.js server
On your_directory_name run, the command npm init –y. After that install express, body-parser, and mongoose.
npm install express body-parser mongoose
_**Create a Model folder and create a file named **_Survey.js
This file includes the schema for the database. Write the following code to create a mongoose schema.

const mongoose = require('mongoose');  
const Schema = mongoose.Schema;  
const surveySchema = new Schema({  
    name: {  
        type: String,  
        default: 'Anonymous'  
    age: {  
        type: String,  
        default: 'Not Specified'  
        type: String,  
        default: 'Not Specified'  
        type: String,  
        default: 'Not Specified'  
        type: String,  
        default: 'Not Specified'  
module.exports = mongoose.model('Survey', surveySchema);

After that,Create a server.js file. Write the following code to start the server at port 3000.

const express = require('express');  
const app = express();  
const bodyParser = require('body-parser');  
const port = 3000;  
const survey = require('./Model/Survey');  
//connect to mongodb  
const mongoose = require('mongoose');  
mongoose.connect('mongodb+srv://<your_mongodb_username>:<your_mongod_password><collection_name>', {useNewUrlParser: true, useUnifiedTopology: true})  
.then(() => console.log('Connected to MongoDB...'))  
.catch(err => console.error('Could not connect to MongoDB...'));  
app.listen(port, () => {  
    console.log(`Example app listening at http://localhost:${port}`);  

Add this code to serve the index.html on running the localhost:3000/

app.get('/', (req, res) => {  
    //respond the static index.html file  
    res.sendFile(__dirname + '/index.html');  

Now create an endpoint that handles the post request of saving the survey data to the MongoDB database.'/survey', async(req, res) => {  
    const obj = new survey({  
        age: req.body.age,  
        sex :,  
        platform: req.body.socialMedia,  
        frequency: req.body.frequency  
        res.status(200).send('Data saved successfully');         
    .catch(err => {  
        res.status(400).send('Unable to save to database');  

To provide all the data, add this code to endpoint “/data”

app.get('/data', async(req, res) => {  
    try {  
        const data = await survey.find();  
    } catch (err) {  
        res.status(500).send('Server Error');  

After that, we need to send the post request from the frontend part so, add this code to index.html.

await fetch('http://localhost:3000/survey',{  
  }).then((res)=>{ //your code here  
  }).catch((err)=>{ //your code here  

Step 3: Deploying to Render Platform
To deploy the application, we need to create a repository in our GitHub. After pushing all the code to the repository. We can log in to the rendering platform with a GitHub account. Click on the new button, list of services will be seen. Click on web service. It provides a free plan for web services. Click on the free plan. A new page appears where you can connect your repository.
You can visit this youtube link for guidelines.
Note*: Sometimes node engine errors might arrive during deployment. Add .nvmrc in your directory with the specific required version of the node engine.
After that, you need to change the endpoint with deployed URL in index.html. Update the endpoint by removing ‘http://localhost:3000/survey’ to render the service URL like

Step 4: Data pipelining using Airflow
We need to install airflow. I have to install airflow in windows(wsl). You can install it by watching this guideline video link.
Under the file directory dags, create a new dag for fetching and cleaning the data Also, create new folder data with a subfolder surveyData as shown in the figure.

dags and data folder

In, we need to import the necessary files first.

import requests  
from airflow import DAG  
from airflow.operators.python import PythonOperator  
from datetime import datetime  
import pandas as pd  
import os  
import csv  
import json

After that, we define a function that performs the workload of fetching the data from MongoDB collection. The response from the URL is in string form so, we need to convert it to a list using a JSON package.

def fetchData():  
        url = “”  
        response = requests.request("GET", url)  
        data= response.text  
        data = json.loads(data)  
        if not os.path.exists(flocation):  
            with open(flocation, 'w', newline='') as f:  
            # Create a csv writer object  
                writer = csv.writer(f)                  
                # Write the header row  
                # Write the data rows  
                for row in data:  
            with open(flocation, 'w', newline='') as f:  
            # Create a csv writer object  
                writer = csv.writer(f)                  
                # Write the header row  
                # Write the response.text rows  
                for row in data:                      
        print("error while fetching")

After that, create a cleanData function workload.

def cleanData():  
        if not os.path.exists(flocation):  
        df= pd.read_csv("yourpath/data/surveyData/surveyData.csv")  
        #remove duplicates  
        df = df.drop('__v', axis=1)  
        df = df.rename(columns={'_id': 'userID'})  
        df= df.drop_duplicates()  
        print("Could not clean the data!!!")

Now create a DAG

with DAG(dag_id="SURVEY-DATA",  
        catchup=False) as dag:      
        fetch = PythonOperator(  
        clean = PythonOperator(  
fetch >> clean

Step 5: To run this you can type the following code in a different terminal.

airflow webserver  
airflow scheduler  

Now you can visit the airflow home page at “localhost:8080”. Select the SURVEY-DATA dag and you trigger it to run. You can view the graph from UI.

After running the dags, the data folder must have two files shown below.

We have successfully deployed the survey form and made a data pipeline using airflow. The collection of data and cleaning can be scheduled at certain intervals of time like hourly which is written in the DAG code above.

Code for Nepal would like to thank DataCamp Donates for providing Sudip, and several other fellows access to DataCamp, to learn and grow.