Commit 43a84214 authored by chenty's avatar chenty

Basically finished all functional stuff.

parent d600219d
# KV2 - Khala
# T1 - CJMT
Khala is a distributed platform prototype for general computing task execution.
CJMT is a distributed platform prototype for general computing task execution.
This platform aims to provide a easy-deploy and easy-use system which organize and make full use of
a cluster of computers where general computing tasks on command line (e.g. program compiling, model simulation, numeric
......@@ -25,20 +25,20 @@ work majorly in docker containerized environment.
#### Deployment
The easiest way of getting and deploying the system is to use pre-built docker image from
[docker hub](https://hub.docker.com/repository/docker/comradestukov/khala).
[docker hub](https://hub.docker.com/repository/docker/comradestukov/cjmt).
```bash
KHALA='comradestukov/khala:v0.1'
docker pull $KHALA
CJMT='comradestukov/cjmt:v0.1'
docker pull $CJMT
```
Then, deploy the first Judicator.
```bash
KHALA='comradestukov/khala:v0.1'
CJMT='comradestukov/cjmt:v0.1'
IP=`ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'`
docker container run -v /var/run/docker.sock:/var/run/docker.sock \
-p 2000:2000 -p 2001:2001 -p 3000:3000 -p 4000:4000 $KHALA judicator \
-p 2000:2000 -p 2001:2001 -p 3000:3000 -p 4000:4000 $CJMT judicator \
--docker-sock=unix:///var/run/docker.sock \
--boot-print-log \
--etcd-print-log \
......@@ -57,10 +57,10 @@ docker container run -v /var/run/docker.sock:/var/run/docker.sock \
After the Judicator is stable, deploy a Gateway.
```bash
KHALA='comradestukov/khala:v0.1'
CJMT='comradestukov/cjmt:v0.1'
IP=`ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'`
docker container run -v /var/run/docker.sock:/var/run/docker.sock \
-p 7000:7000 $KHALA gateway \
-p 7000:7000 $CJMT gateway \
--docker-sock=unix:///var/run/docker.sock \
--boot-print-log \
--etcd-print-log \
......@@ -71,9 +71,9 @@ docker container run -v /var/run/docker.sock:/var/run/docker.sock \
You have to have at least one Executor to execute tasks.
```bash
KHALA='comradestukov/khala:v0.1'
CJMT='comradestukov/cjmt:v0.1'
IP=`ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'`
docker container run -v /var/run/docker.sock:/var/run/docker.sock $KHALA executor \
docker container run -v /var/run/docker.sock:/var/run/docker.sock $CJMT executor \
--docker-sock=unix:///var/run/docker.sock \
--boot-print-log \
--etcd-print-log \
......@@ -85,13 +85,13 @@ You can have more Executors and Gateways using the commands above. If you wish t
the following commands.
```bash
KHALA='comradestukov/khala:v0.1'
CJMT='comradestukov/cjmt:v0.1'
IP=`ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'`
docker container run -v /var/run/docker.sock:/var/run/docker.sock \
--expose 2000 \
--expose 2001 \
--expose 3000 \
--expose 4000 -P $KHALA judicator \
--expose 4000 -P $CJMT judicator \
--docker-sock=unix:///var/run/docker.sock \
--boot-print-log \
--etcd-print-log \
......@@ -132,7 +132,7 @@ char str[100];
int main()
{
cin.getline(str, 100);
cout << "From Khala: " << str << endl;
cout << "From CJMT: " << str << endl;
return 0;
}
```
......
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
attrs = "==19.3.0"
blinker = "==1.4"
certifi = "==2019.11.28"
chardet = "==3.0.4"
click = "==7.0"
coverage = "==5.1"
idna = "==2.8"
importlib-metadata = "==1.6.0"
itsdangerous = "==1.1.0"
jsonschema = "==3.2.0"
lxml = "==4.5.0"
pyrsistent = "==0.16.0"
python-dotenv = "==0.13.0"
requests = "==2.22.0"
six = "==1.14.0"
urllib3 = "==1.25.8"
xmltodict = "==0.12.0"
xmlunittest = "==0.5.0"
zipp = "==3.1.0"
Flask = "==1.1.1"
Flask-Cors = "==3.0.8"
Flask-Login = "==0.5.0"
Flask-Mail = "==0.9.1"
Flask-SQLAlchemy = "==2.4.1"
Jinja2 = "==2.11.1"
MarkupSafe = "==1.1.1"
SQLAlchemy = "==1.3.16"
Werkzeug = "==1.0.0"
[requires]
python_version = "3.7"
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
coverage = "*"
[packages]
requests = "==2.22.0"
blinker = "==1.4"
certifi = "==2019.11.28"
chardet = "==3.0.4"
click = "==7.0"
idna = "==2.8"
itsdangerous = "==1.1.0"
urllib3 = "==1.25.8"
Flask-Mail = "==0.9.1"
Flask = "==1.1.1"
Jinja2 = "==2.11.1"
MarkupSafe = "==1.1.1"
Werkzeug = "==1.0.0"
python-dotenv = "*"
jsonschema = "==3.2.0"
lxml = "==4.5.0"
xmltodict = "==0.12.0"
xmlunittest = "==0.5.0"
Flask-SQLAlchemy = "*"
Flask-Login = "*"
Flask-Cors = "*"
[requires]
python_version = "3.7"
# Interface
#当我们启动程序时,首先被执行的是包含程序实例的脚本,即构造
#文件。
#在.flaskenv文件中 Flask在通过FLASK_APP环境变量定义的模
#块中寻找程序实例。所以在启动程序前,我们需要给.flaskenv中的环境
#变量FLASK_APP重新赋值,这里仅写出包名称即可:
#
#url_for() 函数最简单的用法是以视图函数名,作为参数,返回对应的URL。例如,在当前版本的hello.py 程序中调用url_
#for('index') 得到的结果是/。调用url_for('index', _external=True) 返回的则是绝对地
#址,在这个示例中是http://localhost:5000/。
#token:
#Token是服务端生成的一串字符串,以作客户端进行请求的一个令牌,
#当第一次登录后,服务器生成一个Token便将此Token返回给客户端,
#以后客户端只需带上这个Token前来请求数据即可,无需再次带上用户名和密码。
\ No newline at end of file
This diff is collapsed.
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
[requires]
python_version = "3.7"
# -*- coding: utf-8 -*-
from flask import Flask
from flask_login import LoginManager
from flask_mail import Mail
from flask_sqlalchemy import SQLAlchemy
'''
从构造文件中导入变量时不需要注明构造文件的路径,只需要从包
名称导入,比如导入在构造文件中定义的程序实例app可以使用
from interface import app
'''
app = Flask('interface', template_folder="../webpage")
# CORS(app, supports_credentials=True)
#static_folder='../../webpage/',static_url_path=, static_url_path="../webpage"""
app.config.from_pyfile('settings.py') #在创建程序实例后,使用config对象的from_pyfile()方法即可加
#载配置,传入配置模块的文件名作为参数
app.jinja_env.trim_blocks = True
app.jinja_env.lstrip_blocks = True
login_manager = LoginManager() #Flask-Login uses sessions for authentication.
db = SQLAlchemy(app)
mail = Mail(app)
login_manager.init_app(app)
# CORS(app, supports_credentials=True)
@login_manager.user_loader
def load_user(user_id):
from interface.models import User
return User.query.get(int(user_id))
login_manager.login_view = 'login'
login_manager.login_message = 'Please log in to access this page'
login_manager.login_message_category = 'warning'
login_manager.refresh_view = 're_authenticate'
# login_manager.needs_refresh_message = 'Your custom message'
login_manager.needs_refresh_message_category = 'warning'
'''
current_user是当前用户的代理对象,我们经常借助它来调用User类的方
法和属性,但是当用户未登录时,current_user指向的用户对象是FlaskLogin提供的匿名用户类,而这个类并没有can()方法可供调用
'''
from interface import views, commands
'''
当我们启动程序时,首先被执行的是包含程序实例的脚本,即构造
文件。但注册在程序实例上的各种处理程序均存放在其他脚本中,比如
视图函数存放在views.py中、错误处理函数则存放在errors.py中。如果不
被执行,那么这些视图函数和错误处理函数就不会注册到程序上,那么
程序也无法正常运行。为了让使用程序实例app注册的视图函数,错误
处理函数,自定义命令函数等和程序实例关联起来,我们需要在构造文
件中导入这些模块。因为这些模块也需要从构造文件中导入程序实例,
所以为了避免循环依赖,这些导入语句在构造文件的末尾定义。
'''
'''
你在这里也许会疑惑,既然有了程序实例app对象,为什么还需要
current_app变量。在不同的视图函数中,request对象都表示和视图函数
对应的请求,也就是当前请求(current request)。而程序也会有多个程
序实例的情况,为了能获取对应的程序实例,而不是固定的某一个程序
实例,我们就需要使用current_app变量,
Flask会自动帮我们激活程序上下文:
·当我们使用flask run命令启动程序时
·执行使用@app.cli.command()装饰器注册的flask命令时。
当请求进入时,Flask会自动激活请求上下文,这时我们可以使用
request和session变量。。另外,当请求上下文被激活时,程序上下文也被
自动激活。当请求处理完毕后,请求上下文和程序上下文也会自动销
毁。也就是说,在请求处理时这两者拥有相同的生命周期。
如果你需要在没有激活上下文的情况下使用这些变量,可以手动激
活上下文。程序上下文对象使用app.app_context()获取,我们可以使用with
语句执行上下文操作
或是显式地使用push()方法推送(激活)上下文,在执行完相关
操作时使用pop()方法销毁上下文:
而请求上下文可以通过test_request_context()方法临时创建:
current_app变量只有在程序上下文被激活后
才可以使用
'''
\ No newline at end of file
import os
import click
from interface import app, db
@app.cli.command()
@click.option('--drop', is_flag=True, help='Create after drop.')
def initdb(drop):
"""Initialize the database."""
if drop:
click.echo(os.path.dirname(app.root_path))
click.confirm('This operation will delete the database, do you want to continue?', abort=True)
db.drop_all()
click.echo('Drop tables.')
db.create_all()
click.echo('Initialized database.')
# -*- coding: utf-8 -*-
"""
:author: Zijun Huang
"""
from datetime import datetime
from flask_login import UserMixin
from werkzeug.security import generate_password_hash, check_password_hash
from interface import db
class Message(db.Model, UserMixin):
__tablename__ = "message"
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(20))
body = db.Column(db.String(200))
timestamp = db.Column(db.DateTime, default=datetime.utcnow, index=True)
class User(db.Model, UserMixin):
__tablename__= "user"
id = db.Column(db.Integer, primary_key=True)
username = db.Column(db.String(20), unique=True, index=True)
email = db.Column(db.String(254), unique=True, index=True)
password_hash = db.Column(db.String(128))
# school = db.Column(db.String(50))
#school = db.Column(db.String(30))
mathmodel = db.relationship('Mathmodel', back_populates='user')
def set_password(self, password):
self.password_hash = generate_password_hash(password)
# def set_school(self, school):
# self.school = school
def validate_password(self, password):
return check_password_hash(self.password_hash, password)
class Mathmodel(db.Model, UserMixin):
__tablename__ = "mathmodel"
id = db.Column(db.Integer, primary_key=True)
task_id = db.Column(db.Integer, unique=True)
user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
user = db.relationship('User', back_populates='mathmodel')
blinker==1.4
certifi==2019.11.28
chardet==3.0.4
Click==7.0
coverage==5.0.3
Flask==1.1.1
Flask-Cors==3.0.8
Flask-Login==0.5.0
Flask-Mail==0.9.1
Flask-SQLAlchemy==2.4.1
idna==2.8
itsdangerous==1.1.0
Jinja2==2.11.1
MarkupSafe==1.1.1
python-dotenv==0.11.0
requests==2.22.0
six==1.14.0
SQLAlchemy==1.3.13
urllib3==1.25.8
Werkzeug==1.0.0
# -*- coding: utf-8 -*-
import os
import sys
from interface import app
#在Python中,每一个有效的Python文件(.py)都是模块。每一个包
#含__init__.py文件的文件夹都被视作包,包让你可以使用文件夹来组织
#模块。__init__.py文件通常被称作构造文件,文件可以为空,也可以用
#来放置包的初始化代码。当包或包内的模块被导入时,构造文件将被自
#动执行。
# SQLite URI compatible
WIN = sys.platform.startswith('win')
if WIN:
prefix = 'sqlite:///'
else:
prefix = 'sqlite:////'
dev_db = prefix + os.path.join(os.path.dirname(app.root_path), 'data.db')
#,由于配置文件
#被放到了程序包内,为了定位到位于项目根目录的数据库文件,使用
#os.path.dirname(app.root_path)获取上层目录,app.root_path属性存储
#程序实例所在的路径
SECRET_KEY = os.getenv('SECRET_KEY', 'secret string')
SQLALCHEMY_TRACK_MODIFICATIONS = False
SQLALCHEMY_DATABASE_URI = os.getenv('DATABASE_URI', dev_db)
#数据库URL和密钥都会首先从环境变量获取。
MAIL_SERVER = os.getenv('MAIL_SERVER')
MAIL_PORT = 465
MAIL_USE_SSL = True
MAIL_USERNAME = os.getenv('MAIL_USERNAME')
MAIL_PASSWORD = os.getenv('MAIL_PASSWORD')
MAIL_DEFAULT_SENDER = ('Group Project', MAIL_USERNAME)
# Custom config
#app.config['UPLOAD_PATH'] = os.path.join(app.root_path, 'uploads')
#app.config['ALLOWED_EXTENSIONS'] = ['png', 'jpg', 'jpeg', 'gif']
UPLOAD_PATH = os.path.join(app.root_path, 'uploads')
ALLOWED_EXTENSIONS = ['png', 'jpg', 'jpeg', 'gif']
MAIL_SUBJECT_PREFIX = '[JMT]'
from flask import current_app
from flask import send_from_directory
from sqlalchemy.ext.serializer import Serializer
from interface import app
from interface.models import User
def generate_token(api_user):
'''
生成token
:param api_user:用户id
:return: token
'''
#第一个参数是内部的私钥,这里写在共用的配置信息里了,如果只是测试可以写死
#第二个参数是有效期(秒)
s = Serializer(current_app.config["SECRET_KEY"], expires_in=3600)
# 接收用户id转换与编码
token = s.dumps({"id": api_user}).decode("ascii")
return token
def verify_token(token):
'''
校验token
:param token:
:return: 用户信息 or None
'''
# 参数为私有秘钥,跟上面方法的秘钥保持一致
s = Serializer(current_app.config["SECRET_KEY"])
try:
# 转换为字典
data = s.loads(token)
except Exception:
return None
# 拿到转换后的数据,根据模型类去数据库查询用户信息
user = User.query.get(data["id"])
return user
def allowed_file(filename):
return '.' in filename and \
filename.rsplit('.', 1)[1].lower() in app.config['ALLOWED_EXTENSIONS']
def uploaded_file(filename):
return send_from_directory(app.config['UPLOAD_FOLDER'],
filename)
# -*- coding: utf-8 -*-
"""
:author: Ripley
http://127.0.0.1:5000/
"""
import requests
from flask import flash, redirect, url_for, render_template, request, send_from_directory
from flask import jsonify
from flask import session
from flask_login import current_user
from flask_login import login_user, login_required
from flask_login import logout_user
from interface import app, db
from interface.models import User
from lhs.xmlinterpreter import jsontoxml
from lhs.xmlinterpreter import xmltojson
'''
Merge Front-End
'''
@app.route("/res/<path:path>", methods=["GET","POST"])
def res(path):
return send_from_directory("../webpage" + "/res", path)
@app.route("/<path:page>", methods=["GET"])
def webpage(page):
return render_template(page + ".html", **request.args.to_dict(flat=True))
"""
Home Page
"""
@app.route("/", methods=["GET"])
def index():
return redirect(url_for('webpage', page="index"))
@app.route('/api/register', methods=['GET', 'POST'])
def register():
if request.method == 'GET':
return render_template('register.html')
else:
data = request.get_json()
username = data['username']
email = data['register_email']
password = data['set_password']
#validate email address
if User.query.filter_by(email=email).first():
#abort(400)
#flash('Repeated email!', 'warning')
return jsonify({'register': False, 'register_email':False,'email':True})
# validate username
if User.query.filter_by(username=username).first():
#flash('Repeated Username!','warning')
return jsonify({'register': False, 'register_email':True, 'username':False})
#abort(400)