Compare revisions

**/__pycache__/**

stages:

  - tests

  - deploy

testing:

  image: "python:3.7"

  stage: tests

  before_script:

    - apt-get update -y

    - apt-get install -y libsasl2-dev libldap2-dev libssl-dev

    - python -m pip install -r requirements.txt

  script:

    - python -m pytest tests

deploy:

  stage: deploy

  only:

    - master

  script:

    - chmod +x ./setup_push.sh

    - ./setup_push.sh ssh://dokku@$VM_1:22/$APP_NAME master

    - if [ -z ${VM_2+x} ]; then echo "no VM_2 in this group"; else ./setup_push.sh ssh://dokku@$VM_2:22/$APP_NAME master; fi

web: gunicorn app:app --log-file=-

 No newline at end of file

# Python3 Flask Template

A started template for Flask web apps using **Python 3.8**. This started template includes:

- Dynamic frontend templates (not React)

- Database support

- Imperial LDAP user authentication

- Multiple environments (development and production)

- DoC Materials and Imperial Panopto API functionality

This template is useful if you would like to create a backend REST API, optionally coupled with a simple dynamic frontend.

## Getting started

Once you have created your new app, take a few minutes to look through the files to familiarise yourself with the project structure.

- `app.py` : entry point to the Flask app

- `templates/` : contains the frontend dynamic HTML files

- `static/` : contains the static frontend assets (images, stylesheets and scripts)

- `blueprints/` : contains all the application routes

- `models/` : contains all the database models/schemas

- `database/` : contains the database creation

- `config/` : contains the app settings for the different environments

**The first change you should make** is to set the `APP_NAME` variable in `config/config.py` to whatever you app's name is.

To start the application locally, you can just run `python3 app.py` and this will launch the app on port 5000 (by default).

You will notice a message in the console saying: 

`WARNING: Could not connect to the given database URL!`

To fix this, you should set the environment variable DATABASE_URL accordingly. If you have PostgreSQL running locally, you can use that. Alternatively, you could use SQLite which is much simpler and does not require installation, for example, by running ```export DATABASE_URL="sqlite:///dev.db"```

If you do not want to use a database yet, you can ignore this warning and delete any routes that interact with the database.

If you navigate to `http://localhost:5000`, you will see the response created by the route defined in `blueprints/home.py`.

You will also notice the lines `Environment: production` and `Debug mode: off` when the Flask application starts in the console. To enable debug mode, you must set the environment variable `ENV` to `dev`, ie: `export ENV=dev` (see `config/config.py` for more details on different environments).

## Tutorial 1: Adding a new route

In this template, we have provided several routes in the ```blueprints/``` directory. For example, we have provided all authentication-related routes (such as login and logout) in the ```auth.py```. It is good practice to group your routes logically and create new blueprints if necessary.

To create a new blueprint, create a new file with the name of your blueprint and add:

```python

from flask import Blueprint

my_new_blueprint = Blueprint("new_bp", __name__, url_prefix="/new_bp")

```

You can then add a new route to that blueprint as follows:

```python

@my_new_blueprint.route("/hello")

def hello():

    return "Hello!"

```

Finally and importantly, you must **register the blueprint** with the Flask application. To do this, go to ```app.py``` and register the blueprint in the same style the existing code does so.

Your new route will now be accessible at ```/new_bp/hello```.

## Tutorial 2: Adding database interaction

An important part of web app development is adding database support. If you ticked the "Include DB" option when creating the app, you will have automatically received a PostgreSQL database. You can get the connection string on the dashboard and connecting to the database using it to verify that you do indeed have a database.

For local development, you should either use an SQLite db or install PostgreSQL, and set the ```DATABASE_URL``` environment variable as explained in the Getting Started section.

### Defining database schemas

When working with relational databases, you must define **schemas** (models) for your tables. For example, if you are developing a web app where users can write blog posts, you may want to define a schema for a blog post and a user.

To do this, you can create new schemas/models in the ```models/``` directory and create a new file. In this template, we have an ```Entity``` model which has a primary key ID, a name which is a string and an age which is an int.

For the blog example, let's create a schema for a blog post. Create a file called ```post.py``` and add:

```python

from database.db import db

class Post(db.Model):

    id = db.Column(db.Integer, primary_key=True)

    title = db.Column(db.String(120), unique=False, nullable=False)

    text = db.Column(db.Test, unique=False, nullable=False)

```

You then need to make sure the database will create the relevant tables in the database. Go to ```app.py``` and import your model here:

```python

# Import all your database models just like below:

with app.app_context():

    from models.entity import Entity

    if app.config["SQLALCHEMY_DATABASE_URI"]:

        db.create_all()

```

For more on schemas using SQL-Alchemy, [read here](https://flask-sqlalchemy.palletsprojects.com/en/2.x/models/).

### Querying the database

You can now use your database to create, read, update or delete records using the SQL-Alchemy package. For example, if we want to create a new Post and add it to the database, we can do something like:

```python

from database.db import db

post = db.session.query(Post).filter_by(title="My article").first()

new_post = Post(title="A new article!", text="This is my amazing article, I hope you enjoyed it.")

db.session.add(new_post)

db.session.commit()

```

For more examples, please read around the [SQL-Alchemy package](https://flask-sqlalchemy.palletsprojects.com/en/2.x/).

### Running migrations and updating schemas

To migrate your production database, you should connect to it directly using the connection string available in the PaaS dashboard with a tool like [PgAdmin](https://www.pgadmin.org/) (or connecting to it using your terminal), and run the relevant queries there.

## Tutorial 3: Configuring a test environment

Often, we want to test that our web application is working as expected by writing a series of tests. For a simple example, you can view the code in the ```tests/``` directory.

Unit testing a web app can be difficult, so we can test simple behaviours such as checking that a particular route is working as expected. We can setup a dummy app using a test configuration, as demonstrated in the ```test_home.py``` file. You can add more tests in this directory by making new functions with the ```test_``` prefix.

To run these tests, you can run ```python -m pytest tests``` from the root directory. You should always run these tests before pushing/deploying new code, as they can catch out bugs. In fact, the CI/CD pipeline will run the tests for you, meaning that if a test fails, the pipeline will stop and your code won't be deployed until you fix the code causing the tests to fail.

from .app import app

 No newline at end of file

from flask import Flask, send_from_directory

from database.db import db

from config.config import ENV, get_app_config, get_static_url

# Create and configure our Flask app

app = Flask(__name__, static_url_path=get_static_url())

app.url_map.strict_slashes = False

app.config.from_object(get_app_config())

# Setup authentication

import imperial_ldap

imperial_ldap.init_app(app)

imperial_ldap.set_auth_type(imperial_ldap.config.AuthType.SESSION, redirect_url='auth.login')

# Add integration to the Materials API

import imperial_doc_materials

imperial_doc_materials.init_app(app)

imperial_doc_materials.set_redirect_url('auth.login')

# Setup panopto api

import imperial_panopto

imperial_panopto.init_app(app)

imperial_panopto.set_client_id("<SET YOUR CLIENT ID HERE>")

imperial_panopto.set_client_secret("<SET YOUR CLIENT SECRET HERE>")

app.register_blueprint(imperial_panopto.panopto_api_blueprint)

imperial_panopto.set_panopto_is_dev(ENV != 'prod')

db.init_app(app)

# Import all your database models just like below:

with app.app_context():

    from models.entity import Entity

    if app.config["SQLALCHEMY_DATABASE_URI"]:

        db.create_all()

# Serve all static assets for the frontend

@app.route('/static/<path:path>')

def serve_static_files(path):

    return send_from_directory('static', path)

# Register all routes from the blueprints module

from blueprints.home import home_blueprint

from blueprints.auth import auth_blueprint

app.register_blueprint(home_blueprint)

app.register_blueprint(auth_blueprint)

# Hook any custom Jinja templating functions

from config import CUSTOM_TEMPLATE_FUNCTIONS

app.jinja_env.globals.update(CUSTOM_TEMPLATE_FUNCTIONS)

if __name__ == '__main__':

    app.run()

 No newline at end of file

from flask import Blueprint, render_template, request, redirect, flash

from imperial_ldap.auth import ldap_login, ldap_logout

from config import url_for2

from imperial_doc_materials import materials_login

auth_blueprint = Blueprint("auth", __name__, url_prefix="/auth")

@auth_blueprint.route("/login", methods=["GET", "POST"])

def login():

    if request.method == "GET":

        return render_template("login.html") 

    # Handle post request

    username = request.form.get("username")

    password = request.form.get("password")

    if username and password:

        user = ldap_login(username, password)

        if user:

            # We're logged in here, so now we also login into the Material API

            materials_login(username, password)

            return redirect(url_for2("home.dashboard"))

        else:

            # bad credentials

            flash("Invalid login credentials.")

            return redirect(url_for2("auth.login"))

    return "Please provide a username and password"

@auth_blueprint.route("/logout", methods=["GET"])

def logout():

    ldap_logout()

    return redirect(url_for2("home.index"))

from flask import Blueprint, render_template, request, redirect

from imperial_panopto import panopto_login_required

from config import url_for2

from config.config import ENV, APP_NAME

from database.db import db

from models.entity import Entity

from imperial_ldap.auth import login_required, get_user_from_session

from imperial_doc_materials import using_materials

home_blueprint = Blueprint('home', __name__, url_prefix='/')

@home_blueprint.route('')

def index():

    user = get_user_from_session()  # If there is no user logged, this returns None

    return render_template('index.html', user=user, dev=ENV, appname=APP_NAME)

@home_blueprint.route('/dashboard')

@login_required     # Indicate that the user must be logged in (via LDAP)

@using_materials    # Indicate that we want to access the Materials API

def dashboard(materials_client, user):

    my_courses = materials_client.get_courses_for_year("2122")

    return render_template('dashboard.html', user=user, my_courses=my_courses, dev=ENV, appname=APP_NAME)

@home_blueprint.route('/panopto_example')

@panopto_login_required

def panopto_example(panopto_client):

    video = panopto_client.get_partially_viewed_videos()[0]

    return render_template('panopto_example.html', video=video, iframe=panopto_client.get_iframe_of_video(video.id), dev=ENV, appname=APP_NAME)

@home_blueprint.route('/entities', methods=['GET', 'POST'])

@login_required

def entities(user):

    if request.method == "GET":

        try:

            rows = ""

            for e in db.session.query(Entity).all():

                rows += str(e) + "<br>"

            return rows

        except:

            return "App database error"

    else:

        name = request.form.get('name')

        age = request.form.get('age')

        try:

            db.session.add(Entity(name=name, age=int(age)))

            db.session.commit()

            return redirect(url_for2('.entities'))

        except:

            return "Could not add entity."

# Define any custom Jinja2 functions here

from flask import url_for

from .config import URL_PREFIX

# Get around the routing prefix issue inside the templates

def url_for2(endpoint: str, **kwargs):

    return URL_PREFIX + str(url_for(endpoint, **kwargs))

CUSTOM_TEMPLATE_FUNCTIONS = {

    "url": url_for2,

}

import os

# This should match exactly the name of the app you specified

APP_NAME = "MY_APP_NAME_HERE"

ENV = os.environ.get('ENV', 'prod').lower()

URL_PREFIX = f"/{APP_NAME}" if ENV == 'prod' else ""

# Get the static URL of the app (to get around the production path issue)

def get_static_url():

    if ENV == 'prod':

        return f'/{APP_NAME}/static'

    else:

        return '/static'

# Get the app configuration based on the ENV environment variable (default is prod)

def get_app_config():

    if ENV == 'prod':

        return ProductionConfig()

    else:

        return DevelopmentConfig()

# If you have created a database for this app, the connection string will be automatically

# accessible through the DATABASE_URL environment variable.

def get_db_url(testing=False):

    if testing:

        return 'sqlite://{0}/{1}.db'.format("tmp", "testing")

    url = os.environ.get('DATABASE_URL')

    if url is None:

        print("WARNING: Could not connect to the given database URL!")

    # For PostgreSQL databases, the conn string needs to start with "postgresql"

    if url and url.startswith("postgres://"):

        url = url.replace("postgres://", "postgresql://", 1)

    return url

class CommonConfig:

    APP_NAME = APP_NAME

    URL_PREFIX = URL_PREFIX

    SECRET_KEY = "my-secrefasdfasdfasdfasdt-key"

    SESSION_COOKIE_SECURE = True

    SQLALCHEMY_DATABASE_URI = get_db_url()

    SQLALCHEMY_TRACK_MODIFICATIONS = False

# Flask App settings for production environment

class ProductionConfig(CommonConfig):

    DEBUG = False

    APPLICATION_ROOT = f"/{APP_NAME}"

# Flask App settings for local development enviroment

class DevelopmentConfig(CommonConfig):

    DEBUG = True

# Settings for running tests

class TestConfig(CommonConfig):

    TESTING = True

    SQLALCHEMY_DATABASE_URI = get_db_url(testing=True)

 No newline at end of file

from flask_sqlalchemy import SQLAlchemy

db = SQLAlchemy()

 No newline at end of file

from database.db import db

class Entity(db.Model):

    id = db.Column(db.Integer, primary_key=True)

    name = db.Column(db.String(80), unique=False, nullable=False)

    age = db.Column(db.Integer, unique=False, nullable=False)

    def __repr__(self):

        return f'Entity({self.id}, {self.name}, {self.age})'

Flask

Flask-RESTful

gunicorn

itsdangerous

Jinja2

peewee

requests

pytest

Werkzeug

Flask-SQLAlchemy

SQLAlchemy

psycopg2-binary

pyjwt

python_ldap

requests_oauthlib

https://gitlab.doc.ic.ac.uk/paas-packages/imperial_panopto/-/raw/master/dist/imperial-panopto-0.0.1.tar.gz

https://gitlab.doc.ic.ac.uk/paas-packages/imperial_ldap/-/raw/master/dist/imperial-ldap-0.0.1.tar.gz

https://gitlab.doc.ic.ac.uk/paas-packages/imperial_doc_materials/-/raw/master/dist/imperial-doc-materials-0.0.1.tar.gz

 No newline at end of file

python-3.8.12

#!/bin/bash -e

url=$1

branch=$2

if [ -z "$SSH_PRIVATE_KEY" ]; then

	>&2 echo "Set SSH_PRIVATE_KEY environment variable"

	exit 1

fi

ssh_host=$(echo $url | sed 's/.*@//' | sed 's/[:/].*//')

if [ -z "$ssh_host" ]; then

	>&2 echo "Usage: $0 <user@git.host:project | ssh://user@git.host:port/project> [<branch>]"

	exit 1

fi

ssh_port=

if [[ $url =~ ^ssh://[^/]+:([0-9]+) ]]; then

        ssh_port="-p ${BASH_REMATCH[1]}"

fi

# TODO: skip on multiple runs

mkdir -p ~/.ssh

echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa

chmod 600 ~/.ssh/id_rsa

ssh-keyscan -H $ssh_port "$ssh_host" >> ~/.ssh/known_hosts

git push $url ${CI_COMMIT_SHA:-HEAD}:refs/heads/${branch:-master} $([ -z "$DISABLE_FORCE_PUSH" ] && echo --force)

#content {

    margin-left: 1%;

    margin-right: 1%;

}

 No newline at end of file

html,

body {

  height: 100%;

}

body {

  display: -ms-flexbox;

  display: -webkit-box;

  display: flex;

  -ms-flex-align: center;

  -ms-flex-pack: center;

  -webkit-box-align: center;

  align-items: center;

  -webkit-box-pack: center;

  justify-content: center;

  padding-top: 40px;

  padding-bottom: 40px;

  background-color: #f5f5f5;

}

.form-signin {

  width: 100%;

  max-width: 330px;

  padding: 15px;

  margin: 0 auto;

}

.form-signin .checkbox {

  font-weight: 400;

}

.form-signin .form-control {

  position: relative;

  box-sizing: border-box;

  height: auto;

  padding: 10px;

  font-size: 16px;

}

.form-signin .form-control:focus {

  z-index: 2;

}

.form-signin input[type="email"] {

  margin-bottom: -1px;

  border-bottom-right-radius: 0;

  border-bottom-left-radius: 0;

}

.form-signin input[type="password"] {

  margin-bottom: 10px;

  border-top-left-radius: 0;

  border-top-right-radius: 0;

}

<!doctype html>

<html>

  <head>

    {% block head %}

    <link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/css/bootstrap.min.css" crossorigin="anonymous">

    <link rel="stylesheet" href="{{ url('static', filename='css/base.css') }}" crossorigin="anonymous">

    <title>{% block title %}{% endblock %}</title>

    {% endblock %}

  </head>

  <body>

    <div id="content">

      <!-- Flashed alerts go here --> 

      {% with messages = get_flashed_messages() %}

        {% if messages %}

          <ul class=flashes>

          {% for message in messages %}

            <div class="alert alert-danger alert-dismissible fade show" role="alert">

              <strong>{{ message }}</strong> Please try again with your Imperial username and password.

              <button type="button" class="close" data-dismiss="alert" aria-label="Close">

                <span aria-hidden="true">&times;</span>

              </button>

            </div>

          {% endfor %}

          </ul>

        {% endif %}

      {% endwith %}

      <!-- Body of the web page goes here -->

      {% block content %}{% endblock %}

    </div>

    <script src="https://code.jquery.com/jquery-3.4.1.slim.min.js" crossorigin="anonymous"></script>

    <script src="https://cdn.jsdelivr.net/npm/popper.js@1.16.0/dist/umd/popper.min.js" crossorigin="anonymous"></script>

    <script src="https://stackpath.bootstrapcdn.com/bootstrap/4.4.1/js/bootstrap.min.js" crossorigin="anonymous"></script>

  </body>

</html>

 No newline at end of file

{% extends "base.html" %}

{% block title %}Dashboard{% endblock %}

{% block content %}

    <nav class="navbar navbar-expand-lg navbar-light bg-light">

        <a class="navbar-brand" href="{{ url('home.index') }}">My Web App</a>

        <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarNavAltMarkup" aria-controls="navbarNavAltMarkup" aria-expanded="false" aria-label="Toggle navigation">

          <span class="navbar-toggler-icon"></span>

        </button>

        <div class="collapse navbar-collapse" id="navbarNavAltMarkup">

          <div class="navbar-nav">

            <a class="nav-item nav-link" href="{{ url('home.index') }}">Home <span class="sr-only">(current)</span></a>

            <a class="nav-item nav-link active" href="{{ url('home.dashboard') }}">Dashboard</a>

              {% if dev == "prod" %}

              <a class="nav-item nav-link" href="{{ '/' + appname + '/panopto_example' }}">Panopto Example</a>

              {% else %}

              <a class="nav-item nav-link" href="{{ '/panopto_example' }}">Panopto Example</a>

              {% endif %}

            <a class="nav-item nav-link" href="{{ url('auth.logout') }}">Logout</a>

          </div>

        </div>

    </nav>

    <h1>My Dashboard</h1>

    <h3>Welcome back, {{ user.name }} {{ user.surname }}!</h3>

    <p><strong>Username:</strong> {{ user.username }}</p>

    <p><strong>User Type:</strong> {{ user.title }}</p>

    <br>

    <br>

    <h6>Example CRUD form - Create a record in the database:</h6>

    <form method="post" action="{{ url('home.entities') }}" style="width: 30%;">

        Name:<br>

        <input type="text" name="name" class="form-control">

        <br>

        Age:<br>

        <input type="number" name="age" class="form-control">

        <br>

        <button class="btn btn-lg btn-primary btn-block" type="submit">Create Entry</button>    

      </form>

    <br>

    <a href="{{ url('home.entities') }}">View records</a>

{% endblock %}

 No newline at end of file

{% extends "base.html" %}

{% block title %}Home{% endblock %}

{% block content %}

    <nav class="navbar navbar-expand-lg navbar-light bg-light">

        <a class="navbar-brand" href="{{ url('home.index') }}">My Web App</a>

        <button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#navbarNavAltMarkup" aria-controls="navbarNavAltMarkup" aria-expanded="false" aria-label="Toggle navigation">

          <span class="navbar-toggler-icon"></span>

        </button>

        <div class="collapse navbar-collapse" id="navbarNavAltMarkup">

          <div class="navbar-nav">

            <a class="nav-item nav-link active" href="{{ url('home.index') }}">Home <span class="sr-only">(current)</span></a>

            <a class="nav-item nav-link" href="{{ url('home.dashboard') }}">Dashboard</a>

              {% if dev == "prod" %}

              <a class="nav-item nav-link" href="{{ '/' + appname + '/panopto_example' }}">Panopto Example</a>