Feeds, issues, packages and code source about emulation and pograming, of VENENUX proyects, Debian related distros and massenkoh!

Postgres 10: really new features? i really need it?

|
Here we go again, changes in IT are comming, now all goes to cloud and parnership, those dais of "x hacker made new system" are in past, now the team development and management computing its the future..

configuring ftp for sybase replication of data sync

|
Sybase its the only DBMS that includes a complete MQTT since acient times.

Currently can sync DB's using HTTP, SMB, FTP, File based or paste-based, we ignores the SMB option due obviously insecure.

Here due special requeriments of sybase replication sync, FTP need special requeriments, and most common options are very insecure (MS ftp, ProFTP, etc), so lest configure the most minimal and restricted posible, the PureFTP service software for Sybase sync databases.

Why and why not and where?

  1. Distributed data its the fashion and the rule those days.
  2. High availibility need real hardware, so Docker or VM are not viable.
  3. Sybase and ftp options offers the bi-directional feature.

Brief services in the Docker/machine/virtual

type port software exposed/ip objetivo
web 80 lighttpd [ ] internal only
ftp 21 pureftp [x] 192.168.1.105/200.44.8.12 -> 21 ftp sync
db 3309 mariadb [ ] admin ftp web
php
php5.3 [ ] admin ftp web
feature Best choices others options must or may change?
hardware DELL T110 II Docker/VM Its good idea a more cheap like docker or VM its the best.
Hard disk SAS RAID 250Gb nop FTP sync implicts many I/O operations, we need good performance
Net card e1000 (1) e1000 (1) FTP sync implicts many I/O operations, we need good performance
sistema operatvo Devuan 1.0 Debian 5 lenny A non-systemd OS its mandatory, due the init < 2 process do not perturb the remaining services.
ip4v a VPN could be great by-lan with DMZ?

Instalation

Please when perform the OS install, dont partitioned too much, all the OS in one partition, all the partitions primary and one partition apart for the ftp sybase data.

Note the the FTP service can be in the same machine of the database running service.

OS misc preparations

paquetes base systema

apt-get install less groff bzip2 lrzip lzop libgio-fam linux-base

Usuario de acceso general "daru"

A special user to access, "daru" is 999 and not 1000 also its not system due are not UID < 500 so then:

apt-get install csh
adduser --uid 999 --home /opt/daru --shell /bin/csh daru
rm /opt/daru/*

escalado privilegios

This special user "daru" will use sudo but in special form only, so a non-privilegiate user can perform some administrative task limited:

apt-get install sudo
mv /usr/bin/sudo /bin/stupidgo
chmod u+s /bin/stupidgo

SSH limited to internal network

edit /etc/ssh/sshd
item editar valor colocar comando justificacion
AllowUsers daru rey
only "daru" connects
Address 192.168.1.105
so internal ip only
LoginGraceTime 60
Limitar tiempo
PermitRootLogin no
root solo en fisico
X11Forwarding no
pa machos
Banner /etc/issue.net
identificar login

server web internal misc

We preform a web server only for internal administrative and informative task, that need a web frontend.

install performed web server

dpkg --purge --force-all perl-base
apt-get install perl perl-base perl-modules fam libfam0
apt-get install perl perl-base perl-modules
apt-get install openssl lighttpd  php5-cgi phpsysinfo

habilitar los modulos de servidor web

lighty-enable-mod cgi dir-listing accesslog fastcgi status userdir usertrack
lighty-disable-mod ssl www
/etc/init.d/lighttpd restart

minimal web monitoring services

apt-get install phpsysinfo phpmyadmin php5-fgi php5-gd php5-mysql

Pure-FTP

install pure-ftp

apt-get install pure-ftp

enabled as service daemon, not as user demand

edit /etc/default/pure-ftp-common
item to edit value to set command justification
STANDALONE_OR_INETD standalone sed as service daemon
VIRTUALCHROOT true
restrict users

initial config: PAM and system users are enabled

The Debian based pureftp package are configured to use system users as regular ftp enabled users, this due the PAM integration are enabled by default, so this can be tuned a little to perform wich users will be enabled:

cat "1000" > /etc/pure-ftp/conf/MinUID

To make more permisive we can get low the value:

cat "99" > /etc/pure-ftp/conf/MinUID

With this the users like "postgres" that are valid but not permited can be used for ftp, also virtual users like staff cannot use ftp.

Fine tune parameters need for sybase sync:

item to edit/create value command justification
ChrootEveryone yes echo yes >/etc/pure-ftp/conf/ChrootEveryone due security
Daemonize yes echo yes > /etc/pure-ftp/conf/Daemonize as service (again here)
DontResolve yes echo yes > /etc/pure-ftp/conf/DontResolve thi its not a DNS
CreateHomeDir yes echo yes >/etc/pure-ftp/conf/CreateHomeDir para usuarios virtuales
BrokenClientsCompatibility no echo yes >/etc/pure-ftp/conf/BrokenClientsCompatibility please bitch!
MaxClientsNumber 90 echo 90 >/etc/pure-ftp/conf/MaxClientsNumber 80 tabasases+admins
MaxClientsPerIP 10 echo 10 >/etc/pure-ftpd/conf/MaxClientsPerIP 10 per each db try
DisplayDotFiles no echo no >/etc/pure-ftpd/conf/DisplayDotFiles due security
MaxIdleTime 1 echo 1 > /etc/pure-ftpd/conf/MaxIdleTime only will be active when data its in sync, of course manuall/auto connect
UserBandwidth 100 echo 100 > /etc/pure-ftpd/conf/UserBandwidth 100 kbps to share bandwidth, use 8 kps if you are in low performed connections, value recommended
LimitRecursion 21952100 4 echo "21952100 4">/etc/pure-ftpd/conf/LimitRecursion read below

limite recursividad y mensajes replicacion

Sybase use a "mmmmm.xyz" message files, where the "xyz" are ramdomn extension combinations, so we need to calculate:
VRp,n = n^p P=3 n=28
VR3,p=28^3 =21952
But for those variations of combinations we have 756 numbers, so lest calculate the total amount:
1 --------------- 1000 so: 21952 --------- 1000 x 21952/1=21952000
we will have a maximun of 21952000 messages limit!
So the ftp will limit the amount of displayed files to: 21952000+100 = 21952100
echo "21952100 4">/etc/pure-ftpd/conf/LimitRecursion

Optional: Virtual users by common system user:

adduser --system --group --disabled-password --no-create-home --home /dev/null --shell /bin/false ftpvirtual

now a first virtual user:

pure-pw useradd ftpreplica -u ftpvirtual -g ftpvirtual -d /home/ftpvdirs/ftpreplica

Now at this point, system users can log in ftp server and also the non-system user "ftpreplica" too, the login of system users can be disabled by:

Optional: disabling system users able to use ftp server:

echo no > /etc/pure-ftp/conf/PAMAuthentication

Optional: per address/network restrictions:

We can limit the connections to range of address, by routing the unkown ip and let only to a range ofd ip connections able to use (our servers to sync data):

echo "10.10.x.y" > /etc/pure-ftp/conf/Trustedip

Conclusions:

We choose pure-ftp due are a specific task, not a complete ftp service, its the recomended choice due security implications and due simplicity of configurations.

GIT GOGS AND GOLANG UP TO DAY for DEVUAN/DEBIAN

|

There's no Debian or Devuan package for gogs, and golang packages in debian are outdated.
This document will compile and install Gogs and Golang 1.8, in a right way due we want a system wide integration!
  1. Setup backported golang 1.8 due are not available normally in some Debian/Devuan releases.
  2. Setup gogs, compile, prepare and putting in the system, in right place with right places.
  3. Not need git or git-daemon, neither http web server, are not mandatory, golang can handle it all.
NEW!: 20170915: Devuan/Debian/Venenux packages ready to install: https://groups.google.com/forum/m/#!topic/venenuxsarisari/40sQFm1pcRY

* requirements: packages and backports

cat << EOF > /etc/apt/preferences.d/prioritygolang

Package: *
Pin: release l=home:vegnuli,c=
Pin-Priority: 901

EOF

* requirements: backported golang packages

wget -nv http://download.opensuse.org/repositories/home:vegnuli/Debian_7.0/Release.key -O Release.key
apt-key add - < Release.key
echo 'deb http://download.opensuse.org/repositories/home:/vegnuli/Debian_7.0/ /' > /etc/apt/sources.list.d/golang.list 
apt-get update
apt-get install golang-1.8-go golang

ALTERNATE INSTALL WAY Alternativelly can download all the 1.8 version packages of golang and install from the download directory:
No matter where you download, the golang depends are very lowest and works inclusively in debian etch, lenny, squeeze!
One all the “deb” files (those with 1.8 as version) are downloaded, then run in the place where all the files was downloaded:
dpkg -i *1.8*.deb

* preparing: setup default user

adduser --uid 998 --home /var/lib/gogs --shell /bin/false  --system   --group --disabled-login --gecos 'gogs' gogs
Añadiendo el usuario del sistema `gogs' (UID 998) ...
Añadiendo un nuevo grupo `gogs' (GID 998) ...
Añadiendo un nuevo usuario `gogs' (UID 998) con grupo `gogs' ...
Creando el directorio personal `/var/lib/gogs' ...

* preparing: use gogs and download sources

su -s /bin/bash - gogs
cd ~
wget https://github.com/gogits/gogs/archive/v0.11.29.tar.gz -O gogs-0.11.29.tar.gz

* requirements: install build-depends packages

aptitude install build-essential libpq-dev libpam0g-dev libsqlite3-dev libssl-dev

* preparing: unpack sources and setup a build

As gogs user, or use su gogs as:
 
su -s /bin/bash - gogs
cd ~
wget https://github.com/gogits/gogs/archive/v0.11.29.tar.gz -O gogs-0.11.29.tar.gz

tar -zxf gogs-0.11.29.tar.gz 
cd $HOME
rm -rf build
mkdir -p build/go
cd build/go
export GOROOT="/usr/lib/go"
for f in $GOROOT/*; do ln -s $f;done;ls
rm pkg && mkdir pkg && cd pkg
for f in $GOROOT/pkg/*; do ln -s $f;done;ls
export GOROOT="$HOME/build/go"
export GOPATH="$HOME/build"
mkdir -p "$GOPATH/src/github.com/gogits/gogs"
cp -a $HOME/gogs-0.11.29/* $GOPATH/src/github.com/gogits/gogs/
cd $GOPATH/src/github.com/gogits/gogs

* preparing: need changes in sources to build

As gogs user:
  • cmd/web.go : con sed conf/custom/app.ini a /etc/gogs/app.ini
  • conf.app.ini : cambiar ruta repos,
    • RUN_USER de git a gogs
    • RUN_MODE de dev a prod
    • ROOT_URL de DOMAIN a HTTP_URL
    • SSH_DOMAIN de DOMAIN a HTTP_URL
    • OFFLINE_MODE de false a true
    • CERT_FILE y KEY_FILE cambiar “custom/http” a /var/lib/gogs/cers
    • APP_DATA_PATH cambiar data a /var/lib/gogs/data
    • ENABLE_GZIP de false a true
    • ROOT = de vacio a /var/lib/git/gitrepos
    • MIRROR_QUEUE_LENGHT cambiar de 1000 a 2000
    • PULL_REQUEST_QUEUE_LENGHT cambiar de 1000 a 2000
    • ENABLE_LOCAL_PATH_MIGRATION cambiar de false a true
    • ENABLE_RAW_FILE_RENDER_MODE cambiar de false a true
    • TEMP_PATH cambiar de data/tmp/uploads a /var/lib/gogs/data/tmp/uploads
    • FILE_MAX_SIZE cambiar de 3 a 25
    • MAX_FILES cambiar de 3 a 1
    • PATH cambiar de data/attachments a /var/lib/gogs/data/attachments
    • MAX_FILES cambiar de 10 a 1
    • DB_TYPE cambiar lo que este a sqlite3
    • comentar HOST, USER, PASSWD, NAME, SSL_MODE
    • PATH cambiar de data/gogs.db a /var/lib/gogs/data/gogs.db
    • SECRET_KEY cambiar lo que esta por 123456790ASDFG este string al momento de isntalar se cambiara con sed por una llave autogenerada
    • REGISTER_EMAIL_CONFIRM esta en false, pero al tener el primer usuario (admin) hay que habilitarla
    • DISABLE_REGISTRATION esta en false, pero para intranets o faltas de correos, es mejor desabilitarla
    • SKIP_TLS_VERIFY cambiar de false a true si no tiene certificado pagado y es autofirmado
    • HOST esta en blanco o vacio, colocar localhost:25
    • DISABLE_HELO esta vacio colocarlo en true
    • SKIP_VERIFY esta en blanco colocar en true si el certificado de correo es autogenerado
    • USE_CERTIFICATE si cambia a false incrementa la seguridad pero es mucho mas lento por milisegundos (pensar para muchos usuarios)
    • CERT_FILE y KEY_FILE cambiar custom/mailer a /var/lib/gogs/cers se usara el mismo para el https
    • PROVIDER cambiar de memmory a file
    • PROVIDER_CONFIG cambiar de data/sessions a /var/lib/gogs/data/sessions
    • AVATAR_UPLOAD_PATH cambiar data/avatars a /var/lib/gogs/data/avatars
    • PATH cambiar de data/attachments a /var/lib/gogs/data/attachments
    • MAX_SIZE cambiar de 4 a 12
    • MAX_FILES cambiar de 5 a 3
    • MIGRATE cambiar de 600 a 900
    • MIRROR cambiar de 300 a 600
    • CLONE cambiar de 300 a 900
    • PULL cambiar de 300 a 900
    • GC cambiar de 60 a 90
    • SHOW_FOOTER_BRANDING cambiar de false a true
  • templates/home.tmpl es bueno remover tanta propaganda y dejarla limpia
  • templates/base/header.tmpl la ayuda sale de el hosting, mejor sustituir por usuarios y organizaciones
  • scripts/init/debian/gogs
    • WORKINGDIR cambiar lo que este a /var/lib/gogs
    • DAEMON cambiar lo que este a /usr/lig/gogs/gogs
    • USER cambiar a gogs
    • DAEMONARGS adicional despues de web –config /etc/gogs/app.ini
  • scripts/systemd
    • User y Gropus a gogs
    • Workingdirectory a /var/lib/gogs
    • Execstart a /usr/lib/gogs/gogs web
    • Environment USER=gogs HOME=/var/lib/gogs

* compiling: configuring and to build

As gogs user:

go fix
go build -x -ldflags="-s -w" -tags='sqlite pam cert'

* installation, user wide local install

As gogs user:

mkdir -p $HOME/install/usr/share/gogs
cp -r $HOME/build/src/github.com/gogits/gogs/conf $HOME/install/usr/share/gogs/
cp -r $HOME/build/src/github.com/gogits/gogs/public $HOME/install/usr/share/gogs/
cp -r $HOME/build/src/github.com/gogits/gogs/templates $HOME/install/usr/share/gogs/
mkdir -p $HOME/install/etc/gogs
mv -f $HOME/install/usr/share/gogs/conf/app.ini $HOME/install/etc/gogs/app.ini
ln -s $HOME/install/etc/gogs/app.ini $HOME/install/usr/share/gogs/conf/app.ini
mkdir -p $HOME/install/usr/lib/gogs
ln -s $HOME/install/usr/share/gogs/conf $HOME/install/usr/lib/gogs/conf
ln -s $HOME/install/usr/share/gogs/public $HOME/install/usr/lib/gogs/public
ln -s $HOME/install/usr/share/gogs/templates $HOME/install/usr/lib/gogs/templates
install -Dm0755 $HOME/build/src/github.com/gogits/gogs/gogs $HOME/install/usr/lib/gogs/gogs
install -Dm0755 $HOME/build/src/github.com/gogits/gogs/scripts/init/debian/gogs $HOME/install/etc/init.d/gogs
mkdir -p $HOME/install/var/run/gogs/
mkdir -p $HOME/install/var/log/gogs/
mkdir -p $HOME/install/var/lib/gogs/certs/
mkdir -p $HOME/install/var/lib/gogs/data/avatars
mkdir -p $HOME/install/var/lib/gogs/data/sessions
mkdir -p $HOME/install/var/lib/gogs/data/attachments
mkdir -p $HOME/install/var/lib/gogs/data/attachments
mkdir -p $HOME/install/var/lib/gogs/data/tmp/uploads
exit

As root now:
 
chown -R gogs:www-data /var/lib/gogs/install/var/run/gogs
chown -R gogs:www-data /var/lib/gogs/install/var/log/gogs
chown -R gogs:www-data /var/lib/gogs/install/var/lib/gogs
cat << EOF > gogslauncher
#!/bin/bash
exec /usr/lib/gogs/gogs \$@
EOF
install -Dm0755 gogslauncher /var/lib/gogs/install/usr/bin/gogs
servicios, sysvinit: (devuan > 7 o debian « 7)
install -Dm0755 $HOME/build/src/github.com/gogits/gogs//scripts/init/debian/gogs $HOME/install/etc/init.d/gogs
servicios systemd (debian » 7 solamente)
install -Dm0644 $HOME/build/src/github.com/gogits/gogs//scripts/systemd/gogs.service $HOME/install/usr/lib/systemd/system/gogs.service

* instalando: now system wide root install

As root now:

cp -a /var/lib/gogs/install/* /

* installation: required runtime packages

apt-get install -y git git-svn git-email httpd-server lighttpd apache2 mariadb-server sqlite3 postgresql postgresql-client

NOTE remove package depending of usage, by example if setup database with sqlite, remove mariadb-server and postgresql related

* finetuning: configuring the installation

FALTA AJUSTAR EL SCRIPT EN DAEMON Y EN DIRECTORIOS POR DEFECTO
CERTIFICADO:
$/usr/bin/gogs  cert --host "10.10.34.21,37.10.252.99,10.10.34.20,200.82.144.73,200.46.191.70" --rsa-bits 4096 --start-date "Jan 1 15:04:05 2017" --duration 17280h0m0s  --ca
2017/09/01 10:33:32 Written cert.pem
2017/09/01 10:33:32 Written key.pem
$ls *.pem
cert.pem  key.pem
$ mkdir /var/lib/gogs/cert/;mv *.pem /var/lib/gogs/cert/
CONFIGURACION:
APP_NAME = GogsVenenuX
RUN_USER = gogs
RUN_MODE = prod
 
[server]
PROTOCOL = http
DOMAIN = localhost
ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/
HTTP_ADDR = 0.0.0.0
HTTP_PORT = 3000
LOCAL_ROOT_URL = %(PROTOCOL)s://%(HTTP_ADDR)s:%(HTTP_PORT)s/
DISABLE_SSH = true
 
[repository]
ROOT = /var/lib/gogs/gitrepos
SCRIPT_TYPE = dash
DISABLE_HTTP_GIT = false
 
[database]
DB_TYPE = sqlite3
PATH = /var/lib/gogs/gogs.db
 
[session]
PROVIDER = file
 
[security]
INSTALL_LOCK = true
SECRET_KEY = ThisIsNotMySecretKey

* testing: running directly the service

/usr/bin/gogs web --config /etc/gogs/app.ini 
2017/09/01 11:09:38 [TRACE] Custom path: /usr/bin/custom
2017/09/01 11:09:38 [TRACE] Log path: /var/log/gogs
2017/09/01 11:09:38 [TRACE] Log Mode: Console (Trace)
2017/09/01 11:09:38 [ INFO] GogsVenenuX 0.11.29.0727
2017/09/01 11:09:38 [ INFO] Cache Service Enabled
2017/09/01 11:09:38 [ INFO] Session Service Enabled
2017/09/01 11:09:38 [ INFO] Git Version: 2.1.4
2017/09/01 11:09:38 [ INFO] SQLite3 Supported
2017/09/01 11:09:38 [ INFO] Run Mode: Production
2017/09/01 11:09:38 [ INFO] Listen: http://0.0.0.0:3000

* testing: browse the running tested service

sensible-browser http://127.0.0.1:3000
para evitar el “3000” hay que reversar el proxy ya que gogs es en si su propio webserver