Server setup
From Qi-Hardware
Revision as of 04:53, 26 November 2012 by Wolfgang Spraul (Talk | contribs)
Contents |
History
- 2012-04-25 [ws] removed rss2email and 6 rarely used mailing lists
- 2011-08-28 [ws] updated Indefero (failed, reverted to old version, plan to remove Indefero)
- 2011-07-29 [ws] updated MediaWiki
To Do
- move everything behind SSL only
- remove Indefero, find nice git/svn browser that allows for direct publishing of committed websites
- move Google MX mail exchange and Jabber back to server
- install phpbb (forum), openerp/tryton, rt (request tracker)
- updating an RT ticket should trigger 1-line into #qi-hardware IRC
- Piwik visitors chart on homepage is not cached
- in the Apache config, set '/' to Deny from all, and instead allow access when needed: <Directory /> Order Deny,Allow Deny from All </Directory>
- mailman should add URL to archived post in the footer of each message
- qi-bot should support opt-out from logging (maintain a list of nicks)
- eggdrop log files are 8859-1, not utf-8. add utf-8 support to eggdrop, http://eggwiki.org/Utf-8
- update irclog2html, css support should allow integration like downloads, http://bazaar.launchpad.net/~mgedmin/irclog2html/trunk/annotate/head%3A/CHANGES.txt
- after upgrading to squeeze, bring all -testing packages back to -stable, auto-install security updates
- automatically create OpenWrt packages from wiki pages tagged as scripts (Python, PHP, Lua, etc)
- Marko Knoebl and others are saying that OpenID login in MediaWiki is broken
- find out why css/style.css in downloads cannot use @import to import vector css
- prepare turandot for scalability, cut down/better/more meaningful logging
- turandot VM hangs on reboot, destroy/start on host (libvirt) necessary
- evaluate Planet replacements such as http://www.intertwingly.net/code/venus/ or rawdog
- try to open mailing lists to non-subscribers (how to fight spam?)
- add https to projects server, requested by Sam Geeraerts on list
- Special:WantedTemplates - fill in missing templates
- mailman doesn’t send mails back to sender
- replace use of Google Talk jabber server with something free
- better localization in MediaWiki, maybe separate databases like Wikipedia? pictures & videos also separate?
- exim: SMTP callbacks (CBV), local_sender_callout (used at OM), smtp auth (plain_server, login_server)
- mailman web interface needs to use SSL when entering passwords (currently setup in VirtualHost which cannot do SSL)
- add a web interface to #qi-hardware IRC channel as a MediaWiki extension (not so urgent, webchat.freenode.net is nice)
- setup AWStats
- DNS not fully documented
- after server move to new data center Sep 20, enable IPv6 and find out how to use it
- wiki homepage: add roadmap, what's hot
- write NanoNote software test plan in wiki
- projects homepage should have text and form to make it easy to register a new project
- wiki footer currently says 'All content is dual licensed under CC-BY-SA and GFDL;'. Not precise, switch to latest Wikipedia footer.
- it should not be possible to upload wiki files with licensing set to 'None selected'
base system
virtual machine: 2688 MB, 64-bit debian lenny
*) hostname ~# vi /etc/hostname turandot ~# vi /etc/hosts 127.0.0.1 localhost 88.198.75.224 turandot.qi-hardware.com turandot ~# /etc/init.d/hostname.sh *) verify ~# hostname (should be ‘turandot’) ~# hostname --fqdn (should be ‘turandot.qi-hardware.com’)
*) APT ~# vi /etc/apt/apt.conf.d/50local APT::Default-Release "stable"; APT::Get::Show-Upgraded "true"; APT::Get::Show-Versions "true";
~# vi /etc/network/interfaces auto eth0 iface eth0 inet static address 88.198.75.224-231 netmask 255.255.255.248 gateway 213.239.195.163 pointopoint 213.239.195.163
~# vi /etc/resolv.conf nameserver 213.133.99.99 nameserver 213.133.100.100 nameserver 213.133.98.98
~# vi /boot/grub/menu.lst timeout 0 # kopt=root=/dev/vda1 ro console=tty0 console=ttyS0
~# update-grub
accounts
adduser --disabled-password <user> usermod -a -G www-data <user> ~# vi /etc/ssh/sshd_config AllowUsers ...
- /var/www is the home dir for www-data, but owned by root. When logging in as www-data over ssh, xauth will try for several seconds to write there before giving up. The presence of .Xauthority-c means that xauth will give up immediately, saving the delay.
touch /var/www/.Xauthority-c chown root:root /var/www/.Xauthority-c
DNS
- make sure dig -x 78.46.248.233 returns ‘qi-hardware.com’
- @ TXT “v=spf1 a include:aspmx.googlemail.com ~all”
SSH
~# vi /etc/ssh/sshd_config PasswordAuthentication no
NFS
- client
~# apt-get install nfs-common portmap ~# mkdir /mnt/fidelio-schhist-output ~# vi /etc/fstab fidelio.qi-hardware.com:/home/schhist/output /mnt/fidelio-schhist-output nfs ro 0 0 ~# mount -a
- server
~# apt-get install nfs-kernel-server ~# vi /etc/exports /home/git/repositories fidelio.qi-hardware.com(ro,async,all_squash,no_subtree_check,anongid=107) ~# /etc/init.d/nfs-kernel-server restart
Apache2
~# apt-get install apache2 apache2-doc php5 php5-mysql libapache2-mod-php5 mysql-client
~# vi /etc/apache2/conf.d/qi ServerAdmin webmaster@qi-hardware.com Timeout 20 KeepAliveTimeout 5 MaxClients 70 MaxRequestsPerChild 1000
- Note that the /usr/lib/php5/maxlifetime script reads the gc_maxlifetime value directly from php.ini files, so you cannot use php_value in Apache config files. We increased the garbage cleanup to 3 hours so that long MediaWiki edits will not timeout the login status (although it's better to check 'Remember Me').
~# vi /etc/php5/apache2/php.ini session.gc_maxlifetime = 10800
Exim4
apt-get install exim4-daemon-heavy
- we want exim4-daemon-heavy for SpamAssassin integration
- configuration documentation: /usr/share/doc/exim4-config/README.Debian.gz
- info exim4_files
- note that any domains that receive mail should be resolved with an A record in DNS, not a CNAME. Otherwise some clients (for example qmail) may change the domain name of the email recipient according to the CNAME redirect.
- nice exim cheatsheet: http://bradthemad.org/tech/notes/exim_cheatsheet.php
~# vi /etc/mailname turandot.qi-hardware.com
~# vi /etc/exim4/update-exim4.conf.conf dc_eximconfig_configtype='internet' dc_other_hostnames=`` dc_local_interfaces=`` dc_use_split_config='true'
~# vi /etc/exim4/domains.virtual turandot.qi-hardware.com lists.qi-hardware.com
~# vi /etc/exim4/aliases.virtual @turandot.qi-hardware.com: mailman@lists.en.qi-hardware.com
~# vi /etc/exim4/aliases_rewrite_to discussion@lists.qi-hardware.com: "English Qi Hardware mailing list - support, developers, use cases and fun" <discussion@lists.en.qi-hardware.com> developer@lists.qi-hardware.com: "English Qi Hardware mailing list - support, developers, use cases and fun" <discussion@lists.en.qi-hardware.com> local-es@lists.qi-hardware.com: local-es@lists.en.qi-hardware.com
~# vi /etc/exim4/conf.d/main/000_localmacros MAIN_TLS_ENABLE = true daemon_smtp_ports = smtp : ssmtp : submission tls_on_connect_ports = 465 MAIN_RELAY_NETS = @[] : boheme.qi-hardware.com
~# vi /etc/exim4/conf.d/router/440_exim4-config_virtual
aliases_rewrite_to:
driver = redirect
data = ${lookup{$local_part@$domain}lsearch{/etc/exim4/aliases_rewrite_to}}
headers_remove = to
headers_add = To: ${lookup{$local_part@$domain}lsearch{/etc/exim4/aliases_rewrite_to}}
domains_virtual:
driver = redirect
data = ${lookup{$local_part@$domain}lsearch{/etc/exim4/aliases.virtual}}
domains_virtual_others:
driver = redirect
data = ${lookup{@$domain}lsearch{/etc/exim4/aliases.virtual}}
~# update-exim4.conf *) will generate /var/lib/exim4/config.autogenerated *) verify with ‘exim4 -bV’ and ‘exim4 -bt postmaster@lists.en.qi-hardware.com’
*) certificate *) compare with sample script /usr/share/doc/exim4/example/exim-gencert *) test the certificate with ‘openssl s_server -cert /etc/exim4/exim.crt -key /etc/exim4/exim.crt -www’ then connect with browser to https://IP_OF_SERVER:4433
~# openssl req -newkey rsa:1024 -days 3650 -x509 -nodes -subj '/CN=mail.qi-hardware.com' -set_serial 01 -keyout /etc/exim4/exim.key -out /etc/exim4/exim.crt
~# chmod 0640 /etc/exim4/exim.key /etc/exim4/exim.crt
~# chown root:Debian-exim /etc/exim4/exim.key /etc/exim4/exim.crt
~# /etc/init.d/exim4 restart *) logs are in /var/log/exim4 *) telnet 127.0.0.1 25 and 587 *) openssl s_client -connect 127.0.0.1:465 *) on port 25 and 587, ‘EHLO test’ should show STARTTLS capability *) on port 25 and 587, ‘STARTTLS’ should reply with ‘TLS go ahead’ *) telnet server_ip_from_outside 25 or 587 MAIL From: test@test.com RCPT To: somewhere@somedomain.com should reply with 550 relay not permitted
SpamAssassin
~# apt-get install spamassassin *) documentation in /usr/share/doc/spamassassin *) we haven’t look into adding Pyzor
~# vi /etc/default/spamassassin ENABLED=1 OPTIONS="--create-prefs --max-children 3 --helper-home-dir" CRON=1
~# apt-get install libnet-dns-perl libmail-spf-query-perl libio-socket-ssl-perl ~# sa-update ~# useradd -m -s /bin/false spamassassin ~# /etc/init.d/spamassassin start
~# vi /etc/exim4/conf.d/main/000_localmacros CHECK_DATA_LOCAL_ACL_FILE = /etc/exim4/conf.d/local/acl_check_data
~# vi /etc/exim4/conf.d/local/acl_check_data
accept hosts = +relay_from_hosts
accept condition = ${if >={$message_size}{500k}{yes}{no}}
add_header = X-turandot-SA-Score: over 500k, not checked
warn spam = spamassassin:true
add_header = X-turandot-SA-Score: $spam_score ($spam_bar)
warn spam = spamassassin:true
condition = ${if >{$spam_score_int}{20}{true}{false}}
add_header = X-turandot-SA-Report: $spam_report
warn spam = spamassassin:true
condition = ${if >{$spam_score_int}{50}{true}{false}}
add_header = X-turandot-SA-Status: Yes
deny spam = spamassassin:true
condition = ${if >{$spam_score_int}{70}{true}{false}}
message = This message scored $spam_score spam points on turandot.
~# update-exim4.conf *) verify with ‘exim4 -bV’
~# /etc/init.d/exim4 restart
Mailman
~# apt-get install mailman *) language: English *) http://www.exim.org/howto/mailman21.html *) Debian documentation in /usr/share/doc/mailman/README.Exim4.Debian
~# check_perms -f *) some group errors remain because of symlinks
~# vi /etc/mailman/mm_cfg.py DEFAULT_URL_PATTERN = 'http://%s/mailman/' PRIVATE_ARCHIVE_URL = '/mailman/private' IMAGE_LOGOS = '/mailman/images/' MTA = None VIRTUAL_HOSTS.clear() add_virtualhost('lists.en.qi-hardware.com', 'lists.en.qi-hardware.com') SMTP_MAX_RCPTS = 15 SMTP_MAX_SESSIONS_PER_CONNECTION = 30 ARCHIVE_HTML_SANITIZER = 3 ALLOW_OPEN_SUBSCRIBE = Yes
~# crontab -u list /usr/lib/mailman/cron/crontab.in ~# mmsitepass "site admin password" ~# newlist mailman wolfgang@qi-hardware.com
~# vi /var/lib/mailman/data/sitelist.cfg generic_nonmember_action = 0 # accept, list address is advertised on /cgi-bin/mailman/listinfo require_explicit_destination = 0
~# config_list -i /var/lib/mailman/data/sitelist.cfg mailman ~# echo "wolfgang@qi-hardware.com" | add_members -w n -a n -r - mailman ~# /etc/init.d/mailman start
~# vi /etc/exim4/conf.d/main/000_localmacros
MM_HOME = /var/lib/mailman
MM_UID = list
MM_GID = list
domainlist mm_domains=lists.en.qi-hardware.com
MM_WRAP = MM_HOME/mail/mailman
MM_LISTCHK = MM_HOME/lists/${lc::$local_part}/config.pck
# set MAIN_LOCAL_DOMAINS so that @qi-hardware.com is not considered local and gets looked up via DNS
MAIN_LOCAL_DOMAINS = +mm_domains : lsearch;/etc/exim4/domains.virtual
~# vi /etc/exim4/conf.d/router/450_exim4-config_mailman # after 'begin routers', after the 'system_aliases' router mailman_router: driver = accept domains = +mm_domains require_files = MM_LISTCHK local_part_suffix_optional local_part_suffix = -admin : -bounces : -bounces+* : -confirm : -confirm+* : -join : -leave : -owner : -request : -subscribe : -unsubscribe transport = mailman_transport
~# vi /etc/exim4/conf.d/transport/25_exim4-config_mailman
# after 'begin transports'
mailman_transport:
driver = pipe
command = MM_WRAP \
'${if def:local_part_suffix \
{${sg{$local_part_suffix}{-(\\w+)(\\+.*)?}{\$1}}} \
{post}}' \
$local_part
current_directory = MM_HOME
home_directory = MM_HOME
user = MM_UID
group = MM_GID
~# update-exim4.conf *) will generate /var/lib/exim4/config.autogenerated *) verify with ‘exim4 -bV’ *) test the routing with ‘exim4 -bt _address_’: mailman@lists.en.qi-hardware.com, postmaster@lists.en.qi-hardware.com, root@lists.en.qi-hardware.com, postmaster@qi-hardware.com, root@qi-hardware.com
~# /etc/init.d/exim4 restart
~# vi /etc/apache2/sites-available/en.qi-hardware.com <VirtualHost *:80> ServerName en.qi-hardware.com RedirectMatch permanent ^/mailman/(.*)$ http://lists.en.qi-hardware.com/mailman/$1 RedirectMatch permanent ^/pipermail/(.*)$ http://lists.en.qi-hardware.com/pipermail/$1 </VirtualHost>
~# vi /etc/apache2/sites-available/lists.en.qi-hardware.com <VirtualHost *:80> ServerName lists.qi-hardware.com RedirectMatch /cgi-bin/mailman/listinfo/announce$ http://lists.en.qi-hardware.com/mailman/listinfo/announce RedirectMatch permanent (.*)$ http://lists.en.qi-hardware.com/mailman/listinfo </VirtualHost> <VirtualHost *:80> ServerName lists.en.qi-hardware.com DocumentRoot /srv/www/lists.en.qi-hardware.com <Directory /srv/www/lists.en.qi-hardware.com> Options None AllowOverride None DirectoryIndex index.php </Directory> ScriptAlias /mailman/admin /usr/lib/cgi-bin/mailman/admin ScriptAlias /mailman/admindb /usr/lib/cgi-bin/mailman/admindb ScriptAlias /mailman/confirm /usr/lib/cgi-bin/mailman/confirm ScriptAlias /mailman/create /usr/lib/cgi-bin/mailman/create ScriptAlias /mailman/edithtml /usr/lib/cgi-bin/mailman/edithtml ScriptAlias /mailman/listinfo /usr/lib/cgi-bin/mailman/listinfo ScriptAlias /mailman/options /usr/lib/cgi-bin/mailman/options ScriptAlias /mailman/private /usr/lib/cgi-bin/mailman/private ScriptAlias /mailman/rmlist /usr/lib/cgi-bin/mailman/rmlist ScriptAlias /mailman/roster /usr/lib/cgi-bin/mailman/roster ScriptAlias /mailman/subscribe /usr/lib/cgi-bin/mailman/subscribe Alias /mailman/images/ /usr/share/images/mailman/ Alias /pipermail/ /var/lib/mailman/archives/public/ <Directory /usr/lib/cgi-bin/mailman/> AllowOverride None Options ExecCGI AddHandler cgi-script .cgi </Directory> <Directory /usr/share/images/mailman/> AllowOverride None </Directory> <Directory /var/lib/mailman/archives/public/> AllowOverride None Options Indexes FollowSymlinks </Directory> </VirtualHost>
~# vi /srv/www/lists.en.qi-hardware.com/index.php
<?php header("location: http://lists.en.qi-hardware.com/mailman/listinfo");?>
~# vi /srv/www/lists.en.qi-hardware.com/mailman/index.php
<?php header("location: http://lists.en.qi-hardware.com/mailman/listinfo");?>
~# a2ensite lists.en.qi-hardware.com ~# /etc/init.d/apache2 reload
- CSS
* compare with https://admin-trac.openmoko.org/trac/browser/trunk/lists.openmoko.org * edit /etc/mailman/{qi_html_head.html,qi_html_body_top.html,qi_html_body_bottom.html} * note similarities to downloads setup in /srv/www/downloads.qi-hardware.com/{css/style.css,HEADER.shtml,FOOTER.shtml} * edit /etc/mailman/en/{listinfo.html,options.html,private.html,roster.html,subscribe.html}, add <mm-qi-...> tags * edit /etc/mailman/en/{archidxfoot.html,archidxhead.html,archtoc.html,archtocnombox.html,article.html}, add %(qi-...)s tags * /etc/mailman/en/admlogin.html parsing broken so not patched, URL is /mailman/admin/_listname_ * /etc/mailman/en/emptyarchive.html not used, so not patched * To customize templates for one particular list, copy the template file to
/var/lib/mailman/lists/<particular list>/en/<template file>.html
~# patch /var/lib/mailman/Mailman/Cgi/htmlformat.py
@@ -317,6 +317,7 @@
if self.title:
output.append('%s<TITLE>%s</TITLE>' % (tab, self.title))
+ output.append(open('/etc/mailman/qi_html_head.html').read())
output.append('%s</HEAD>' % tab)
@@ -332,9 +333,11 @@
output.append('dir="%s">' % direction)
+ output.append(open('/etc/mailman/qi_html_body_top.html').read())
# Always do this...
output.append(Container.Format(self, indent))
if not self.suppress_head:
+ output.append(open('/etc/mailman/qi_html_body_bottom.html').read())
output.append('%s</BODY>' % tab)
~# patch /var/lib/mailman/Mailman/Cgi/HTMLFormatter.py
@@ -401,6 +401,9 @@
'<mm-list-langs>' : listlangs,
+ '<mm-qi-html-head>' : open('/etc/mailman/qi_html_head.html').read(),
+ '<mm-qi-html-body-top>' : open('/etc/mailman/qi_html_body_top.html').read(),
+ '<mm-qi-html-body-bottom>' : open('/etc/mailman/qi_html_body_bottom.html').read(),
}
if mm_cfg.IMAGE_LOGOS:
~# patch /var/lib/mailman/Mailman/Archiver/HyperArch.py
@@ -478,6 +478,9 @@
d['encoding'] =
+ d['qi-html-head'] = open('/etc/mailman/qi_html_head.html').read()
+ d['qi-html-body-top'] = open('/etc/mailman/qi_html_body_top.html').read()
+ d['qi-html-body-bottom'] = open('/etc/mailman/qi_html_body_bottom.html').read()
finally:
i18n.set_translation(otrans)
@@ -689,6 +692,7 @@
"listname": html_quote(mlist.real_name, self.lang),
+ 'qi-html-body-bottom' : open('/etc/mailman/qi_html_body_bottom.html').read(),
}
i = {"thread": _("thread"),
@@ -725,6 +729,8 @@
"size": self.size,
+ 'qi-html-head' : open('/etc/mailman/qi_html_head.html').read(),
+ 'qi-html-body-top' : open('/etc/mailman/qi_html_body_top.html').read(),
}
i = {"thread": _("thread"),
@@ -759,6 +765,9 @@
'meta': ,
+ 'qi-html-head' : open('/etc/mailman/qi_html_head.html').read(),
+ 'qi-html-body-top' : open('/etc/mailman/qi_html_body_top.html').read(),
+ 'qi-html-body-bottom' : open('/etc/mailman/qi_html_body_bottom.html').read(),
}
# Avoid i18n side-effects
otrans = i18n.get_translation()
After updating templates, run “/var/lib/mailman/bin/mailmanctl restart” to flush template caches. This will also verify and compile the Python scripts.
- Mailing Lists
- data files are in /var/lib/mailman/archives/private/_listname_.mbox/_listname_.mbox
use ‘/var/lib/mailman/bin/arch --wipe _listname_’ to recreate the list archive
- merge two mbox: cat mbox_1 >> mbox_2
- split mbox into files: formail -ds sh -c 'cat > msg.$FILENO' < mbox_file
- create mbox out of files: for i in maildir/*; do formail < "$i" >> mbox; done
- to move an existing list to a new domain, use fix_url.py (to update hidden web_page_url attribute)
new_domain.com will be looked up in the virtual hosts from mm_cfg.py
bin/withlist -l -r fix_url listname -u new_domain.com
bin/withlist -l -a -r fix_url -- [fix_url options like -u]
- or use withlist directly
withlist -l list_name >>> m.web_page_url current web_page_url displayed >>> m.web_page_url='http://lists.en.qi-hardware.com/mailman/' >>> m.host_name >>> m.Save() >>> quit()
- list creation
~# newlist _listname_ mailman@lists.en.qi-hardware.com ~# config_list -o cfg _listname_ ~# vi cfg …edit config options… ~# config_list -i cfg _listname_
configuration for announce@lists.en.qi-hardware.com
real_name = 'announce' description = 'Copyleft Hardware Announcements' subject_prefix = '[Qi Announce]‘ reply_goes_to_list = 2 # explicit address reply_to_address = discussion@lists.en.qi-hardware.com include_list_post_header = 0 msg_footer = “”"__________________ Qi Hardware Announcements Subscribe or Unsubscribe: %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s”"” digest_footer = “”"__________________ Qi Hardware Announcements Subscribe or Unsubscribe: %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s”"” default_member_moderation = 1 # yes generic_nonmember_action = 2 # Reject forward_auto_discards = 0 # no
configuration for discussion@lists.en.qi-hardware.com
real_name = 'discussion' description = 'English Qi Hardware mailing list - support, developers, use cases and fun' subject_prefix = ' ' max_message_size = 200 msg_footer = """__________________ Qi Hardware Discussion List Mail to list (members only): %(real_name)s@%(host_name)s Subscribe or Unsubscribe: %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s""" digest_footer = """__________________ Qi Hardware Discussion Digest Mail to list (members only): %(real_name)s@%(host_name)s Subscribe or Unsubscribe: %(web_page_url)slistinfo%(cgiext)s/%(_internal_name)s""" generic_nonmember_action = 1 # hold
downloads.qi-hardware.com
autoindex documentation: http://httpd.apache.org/docs/2.2/mod/mod_autoindex.html
tbd: IndexHeadInsert not possible with Apache 2.2.9, wait for newer version
~# ln -s /etc/apache2/mods-available/{dav.load,dav_fs.load,dav_fs.conf} /etc/apache2/mods-enabled
~# vi /etc/apache2/sites-available/downloads.qi-hardware.com <VirtualHost *:80> ServerName downloads.qi-hardware.com DocumentRoot /srv/www/downloads.qi-hardware.com RedirectMatch permanent ^/software/images/NanoNote/Ben/reflash_ben.sh$ http://projects.qi-hardware.com/p/openwrt-xburst/source/file/master/data/qi_lb60/scripts/reflash_ben.sh RedirectMatch permanent ^/software/images/Ben_NanoNote_2GB_NAND/reflash_ben.sh$ http://projects.qi-hardware.com/p/openwrt-xburst/source/file/master/data/qi_lb60/scripts/reflash_ben.sh <Directory /srv/www/downloads.qi-hardware.com> DAV On AuthType Basic AuthName 'downloads.qi-hardware.com' AuthUserFile /etc/apache2/downloads.passwd <Limitexcept GET HEAD PROPFIND OPTIONS REPORT> Require valid-user </Limitexcept> </Directory> </VirtualHost>
- Add user accounts with: htpasswd /etc/apache2/downloads.passwd <user>
~# ln -s /etc/apache2/sites-available/downloads.qi-hardware.com /etc/apache2/sites-enabled ~# a2enmod include ~# mkdir /srv/www/downloads.qi-hardware.com ~# chown www-data.www-data /src/www/downloads.qi-hardware.com
~# vi /srv/www/downloads.qi-hardware.com/.htaccess IndexIgnore HEADER.shtml FOOTER.shtml favicon.ico robots.txt IndexOptions +HTMLTable +FancyIndexing +DescriptionWidth=* IndexStyleSheet /css/style.css HeaderName /HEADER.shtml ReadmeName /FOOTER.shtml AddDescription "..." folder_name
~# vi /etc/cron.d/ls-lR 47 2 * * * www-data cd /srv/www/downloads.qi-hardware.com && find * -path FOOTER.shtml -o -path HEADER.shtml -o -path favicon.ico -o -path robots.txt -o -path ls-lR.txt -o -name .htaccess -o -name .DS_Store -o -type f -fprintf ls-lR.txt '\%8s \%TY-\%Tm-\%Td \%p\n'
Planet
~# apt-get install planet
- HTML symbol #160 (nbsp) is confusing the planet. It will first be converted to a 0xA0 character and then later the utf-8 to Unicode conversion chokes on the non-utf8 0xA0 value. We just replace #160 with a normal space as a workaround.
~# patch -d /usr/lib/pymodules/python2.5/planet -p0 <
(in class _BaseHTMLProcessor:feed())
--- feedparser.py.orig 2010-10-05 21:08:38.000000000 -0400
+++ feedparser.py 2010-10-05 21:10:24.000000000 -0400
@@ -1498,6 +1498,7 @@
data = data.replace('& #39;', "'")
data = data.replace('& #34;', '"')
+ data = data.replace('& #160;', ' ')
if self.encoding and type(data) == type(u):
data = data.encode(self.encoding)
~# cp /usr/share/doc/planet/examples/planet.css /var/lib/planet/www/planet.css ~# cp /usr/share/planet/templates/fancy.html.tmpl /var/lib/planet/www/index.html.tmpl
~# vi /var/lib/planet/www/index.html.tmpl +:insert /etc/mailman/qi_html_head.html: </head> <body> +:insert /etc/mailman/qi_html_body_top.html: <h1><TMPL_VAR name></h1> ... -<div class="sidebar"> -<img src="images/logo.png" width="136" height="136" alt=""> - -<h2>Subscriptions</h2> +:insert modified /etc/mailman/qi_html_body_bottom.html: +<!-- interactive --> +<div class="portal" id='p-interactive'> +<h5>Subscribe To The Planet</h5> +<div class="body"> +<ul class="feed_list"> + <li><a href="atom.xml" class="feed"><img src="images/atom.png" width="80" height="15" alt="Atom Feed"></a></li> + <li><a href="rss20.xml" class="feed"><img src="images/rss20.png" width="80" height="15" alt="RSS 2.0 Feed"></a></li> + <li><a href="rss10.xml" class="feed"><img src="images/rss10.png" width="80" heigh="15" alt="RSS 1.0 Feed"></a></li> + <li><a href="foafroll.xml" class="feed"><img src="images/foaf.png" width="80" height="15" alt="FOAF Subscriptions"></a></li> + <li><a href="opml.xml" class="feed"><img src="images/opml.png" width="80" height="15" alt="OPML Subscriptions"></a></li> +</ul> +<ul><li> +Or subscribe to the <a href="http://lists.en.qi-hardware.com/mailman/listinfo">rss2email feed</a> in <a href="http://lists.en.qi-hardware.com/mailman/listinfo/planet-log-text">text version</a> or <a href="http://lists.en.qi-hardware.com/mailman/listinfo/planet-log-html">html version</a>. +</li></ul> +<br> +</div> + +<h5>Planet Reads From</h5> +<div class="body"> <ul> <TMPL_LOOP Channels> <li> @@ -99,27 +158,48 @@ </ul> <p> -<strong>Last updated:</strong><br> -<TMPL_VAR date><br> -<em>All times are UTC.</em><br> -<br> -Powered by:<br> -<a href="http://www.planetplanet.org/"><img src="images/planet.png" width="80" height="15" alt="Planet" border="0"></a> -</p> - -<p> -<h2>Planetarium:</h2> <ul> -<li><a href="http://www.planetapache.org/">Planet Apache</a></li> -<li><a href="http://planet.debian.net/">Planet Debian</a></li> -<li><a href="http://planet.freedesktop.org/">Planet freedesktop.org</a></li> -<li><a href="http://planet.gnome.org/">Planet GNOME</a></li> -<li><a href="http://planetsun.org/">Planet Sun</a></li> -<li><a href="http://fedora.linux.duke.edu/fedorapeople/">Fedora People</a></li> -<li><a href="http://www.planetplanet.org/">more...</a></li> +<li><strong>Last updated:</strong></li> +<li><TMPL_VAR date></li> +<li><em>All times are UTC.</em></li> +<li>Copyright of this syndicated feed is with the respective authors.</li> +<li>Powered by: <a href="http://www.planetplanet.org/"><img src="images/planet.png" width="80" height="15" alt="Planet" border="0"></a></li> </ul> </p> </div> +<br> + + <h5>interactive</h5> + + <div class="body"> + <ul> + <li id="n-About-Qi"><a href="http://en.qi-hardware.com/wiki/Qi:About">About Qi</a></li> + <li id="n-Community-portal"><a href="http://en.qi-hardware.com/wiki/Qi:Community_portal">Community portal</a></li> + <li id="n-recentchanges"><a href="http://en.qi-hardware.com/wiki/Special:RecentChanges" title="The list of recent changes in the wiki [r]" accesskey="r">Recent changes</a></li> + <li id="n-Contact-Qi"><a href="http://en.qi-hardware.com/wiki/Qi:Contact_us">Contact Qi</a></li> + <li id="n-help"><a href="http://en.qi-hardware.com/wiki/Help:Contents" title="The place to find out">Help</a></li> + </ul> + </div> +</div> <!-- /interactive --> + +</div> <!-- /panel --> + :rest of /etc/mailman/qi_html_body_bottom.html (footer): </html>
~# vi /etc/planet.conf name=copyleft hardware planet link=http://en.qi-hardware.com/planet/ owner_name=Qi Webmaster owner_email=webmaster@qi-hardware.com template_files=/var/lib/planet/www/index.html.tmpl ... # add feeds at end
~# vi /etc/cron.d/planet */30 * * * * ...
~# vi /etc/apache2/sites-available/en.qi-hardware.com Alias /planet /var/lib/planet/www <Directory /var/lib/planet/www> Order Deny,Allow Allow from all Options None AllowOverride None </Directory>
~# vi /etc/apache2/sites-available/planet.qi-hardware.com <VirtualHost *:80> ServerName planet.qi-hardware.com RedirectMatch permanent ^/$ http://en.qi-hardware.com/planet/ RedirectMatch permanent ^/atom.xml$ http://en.qi-hardware.com/planet/atom.xml RedirectMatch permanent ^/rss20.xml$ http://en.qi-hardware.com/planet/rss20.xml RedirectMatch permanent ^/rss10.xml$ http://en.qi-hardware.com/planet/rss10.xml RedirectMatch permanent ^/foafroll.xml$ http://en.qi-hardware.com/planet/foafroll.xml RedirectMatch permanent ^/opml.xml$ http://en.qi-hardware.com/planet/opml.xml </VirtualHost> ~# a2ensite planet.qi-hardware.com
MySQL
~# apt-get install mysql-server
~# vi /root/.my.cnf [0600] [client] password=_pwd_
~# vi /etc/mysql/conf.d/qi.cnf # used mysqltuner and /etc/init.d/mysql status to optimize values [mysqld] table_cache = 200 thread_concurrency = 4 query_cache_limit = 32M query_cache_size = 128M join_buffer_size = 1M log_slow_queries = /var/log/mysql/mysql-slow.log long_query_time = 2 tmp_table_size = 192M max_heap_table_size = 192M innodb_buffer_pool_size = 256M wait_timeout = 60
~# mysql -e 'create user 'drupal'@'localhost' identified by '_pwd_';' ~# mysql -e 'grant all on _db_name_.* to 'drupal'@'localhost';'
- changing a password
~# mysql -e 'set password for 'drupal'@'localhost' = PASSWORD('_new_password_');'
- moving a database to a new server:
src: mysqldump db_name > db_name.sql dest: mysql -e ‘create database db_name;’ dest: mysql db_name < db_name.sql *) Note that this did not work for Chinese characters in database fields, maybe some UTF-8 problem? *) --all-databases dumps all dbs, --xml dumps in xml format
MediaWiki
~# mkdir -p /srv/www/en/w
~# cd /srv/www/en/w
~# svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/phase3 .
*) update with 'svn update', then run 'php update.php' in maintenance dir
*) backup db with mysqldump wikidb > wikidb.sql
- patch svg uploading to allow HTML and JavaScript inside svg
~# patch -p0 <
--- UploadBase.php
+++ /srv/www/en/w/includes/upload/UploadBase.php
@@ -374,7 +374,7 @@
if( $this->mFinalExtension == 'svg' || $mime == 'image/svg+xml' ) {
- if( $this->detectScriptInSvg( $this->mTempPath ) ) {
+ if(0 && $this->detectScriptInSvg( $this->mTempPath ) ) {
return array( 'uploadscripted' );
*) for extensions from the MediaWiki SVN ~# cd /srv/www/en/w/extensions ~# grep extensions ../LocalSettings.php ~# svn co http://svn.wikimedia.org/svnroot/mediawiki/trunk/extensions/{ext1,ext2,ext3} *) update with 'svn update *', then run 'php update.php' in maintenance dir *) to list installed extensions see http://en.qi-hardware.com/wiki/Special:Version *) all remaining documentation about extension specific prerequisites is in LocalSettings.php
- extension intersection (DynamicPageList) needs a patch to add 'random' sort order, check here
~# patch -p0 <
--- DynamicPageList.php
+++ /srv/www/en/w/extension/intersection/DynamicPageList.php
@@ -243,6 +243,9 @@
case 'ordermethod':
switch ( $sArg ) {
+ case 'random':
+ $orderMethod = 'random';
+ break;
@@ -487,6 +490,9 @@
switch ( $sOrderMethod ) {
+ case 'random':
+ $sqlSort = 'RAND()';
+ break;
- extension OggHandler needs a patch to fix a bug, and a currently unreleased oggvideotools >= 0.9
~# patch -p0 <
--- OggHandler_body.php
+++ /srv/www/en/w/extension/intersection/OggHandler_body.php
@@ -26,7 +26,6 @@
if ( $name == 'thumbtime' ) {
- $length = $this->getLength( $image );
$time = $this->parseTimeString( $value );
* run on build machine
~# svn co https://oggvideotools.svn.sourceforge.net/svnroot/oggvideotools oggvideotools
~# cd oggvideotools/trunk
~# ./autogen.sh
~# scp build/src/liboggvideotools.so root@turandot.qi-hardware.com:/usr/local/lib
~# scp build/src/ogg* root@turandot.qi-hardware.com:/usr/local/bin
~# ssh root@turandot.qi-hardware.com ldconfig
~# vi /srv/www/en/w/LocalSettings.php
$wgOggThumbLocation = '/usr/local/bin/oggThumb';
- extension Math
~# apt-get install tetex-bin tetex-extra latex-cjk-all gs-gpl imagemagick dvipng ocaml ~# cd /srv/www/en/w/extensions/Math/math ~# make
- extension PdfHandler
* cmap-adobe packages needed to create thumbnails for some PDFs ~# apt-get install ghostscript cmap-adobe-gb1 cmap-adobe-japan1
*) SVG apt-get install libwmf-bin librsvg2-bin ~# vi /srv/www/en/w/LocalSettings.php $wgAllowTitlesInSVG = true; $wgFileExtensions[] = 'svg'; # we use Inkscape to edit svgs so it's best to also use it for thumbnails # apt-get install inkscape # Download dustismo.zip from http://www.dafont.com/dustismo.font # mkdir /usr/share/fonts/truetype/dustismo # unzip -d /usr/share/fonts/truetype/dustismo dustismo.zip # fc-cache # *) following dirs needed for inkscape to run under the www-data user # mkdir /var/www/.gnome2 # mkdir -p /var/www/.config/inkscape # chown www-data.www-data /var/www/.config/inkscape $wgSVGConverter = 'inkscape';
*) install extensions that are not in the MediaWiki SVN into /srv/www/mediawiki-ext *) All downloaded files are backed up in /srv/install-archive VisualMathCaptcha pChart4mw *) download pChart.1.27d.rar from pchart.sourceforge.net, install to /srv/www/mediawiki-ext/pChart4mw/pChart SecureHTML StubManager ParserFunctionsHelper (GraphViz uninstalled, nobody used it, not in MediaWiki SVN, problems in 1.16) (apt-get install graphviz) (SiteGraph uninstalled, nobody used it, not in MediaWiki SVN, problems in 1.16) (GnuplotBasic not installed, does not look secure enough - needs gnuplot) (WikiTeX not installed, way too complicated) (PdfExport uninstalled, did not include images - needs htmldoc)
*) locally edited files /srv/www/en/w/images [dir] /srv/www/en/w/LocalSettings.php ~# patch -p0 < +++ /srv/www/en/w/includes/mime.types @@ -159,3 +159,4 @@ application/x-opc+zip docx dotx docm dotm potx ppsx pptx ppam pptm potm ppsm xlsx xltx xlsm xltm xlam xlsb dwfx xps +text/plain dxf +application/x-gzip ae
*) locally edited 'system' files via the web interface /wiki/MediaWiki:Common.css /wiki/MediaWiki:Vector.css /wiki/MediaWiki:Sidebar
- VisualMathCaptcha needs a patch
~# patch -p0 <
--- VisualMathCaptcha.body.php 2008-04-17 16:16:10.000000000 -0400
+++ /srv/www/mediawiki-ext/VisualMathCaptcha/VisualMathCaptcha.body.php 2010-10-02 07:55:23.000000000 -0400
@@ -13,7 +13,7 @@
function __construct() {
- SpecialPage::SpecialPage( 'VisualMathCaptcha', , true );
+ parent::__construct( 'VisualMathCaptcha', , true );
}
~# vi /etc/apache2/sites-available/en.qi-hardware.com <VirtualHost *:80> ServerName qi-hardware.com ServerAlias qi-inside.org www.qi-inside.org RedirectMatch permanent (.*)$ http://en.qi-hardware.com/wiki/Main_Page </VirtualHost> <VirtualHost *:80> ServerName qi-hw.com RedirectMatch permanent ^/f/atom$ http://en.qi-hardware.com/feed/atom.xml RedirectMatch permanent ^/f/rss$ http://en.qi-hardware.com/feed/rss20.xml RedirectMatch permanent .* http://en.qi-hardware.com/wiki/Main_Page </VirtualHost> <VirtualHost *:80> ServerName www.qi-hardware.com RedirectMatch permanent ^/$ http://en.qi-hardware.com/wiki/Main_Page RedirectMatch permanent ^/products/ben-nanonote$ http://sharism.cc/products/ben-nanonote/ RedirectMatch permanent ^/products/ben-nanonote/$ http://sharism.cc/products/ben-nanonote/ RedirectMatch permanent ^/products/ya-nanonote$ http://en.qi-hardware.com/wiki/Ya_NanoNote_Specs RedirectMatch permanent ^/products/ya-nanonote/$ http://en.qi-hardware.com/wiki/Ya_NanoNote_Specs RedirectMatch permanent ^/products/mu-nanonote$ http://en.qi-hardware.com/wiki/Mu_NanoNote_Specs RedirectMatch permanent ^/products/mu-nanonote/$ http://en.qi-hardware.com/wiki/Mu_NanoNote_Specs RedirectMatch permanent ^/products/guo-nanonote$ http://en.qi-hardware.com/wiki/Guo_NanoNote_Specs RedirectMatch permanent ^/products/guo-nanonote/$ http://en.qi-hardware.com/wiki/Guo_NanoNote_Specs RedirectMatch permanent ^/feed/(.*)$ http://sharism.cc/feed/$1 RedirectMatch permanent ^/blog/(.*)$ http://sharism.cc/blog/$1 RedirectMatch permanent ^/category/(.*)$ http://sharism.cc/category/$1 RedirectMatch permanent ^/(.*)$ http://en.qi-hardware.com/wiki/Main_Page </VirtualHost> <VirtualHost *:80> ServerName en.qi-hardware.com DocumentRoot /srv/www/en Alias /wiki /srv/www/en/w/index.php <Directory /srv/www/en> Options None # Note that because we set AllowOverride to None, all .htaccess files # in the MediaWiki sources or extension sources need to be included here # in the Apache configuration. Verify from time to time that the .htaccess # files and the config here are in sync. AllowOverride None DirectoryIndex index.php php_value upload_max_filesize 2000M php_value post_max_size 4000M </Directory> # .htaccess files # We enable outside access to extensions one by one as needed. Check the error_log # to see whether additional extensions require HTTP access. <Directory ~ /srv/www/en/w/(cache|maintenance|serialized|includes|languages|math|extensions)> deny from all </Directory> <Directory ~ /srv/www/en/w/(extensions/LiquidThreads|OggHandler|OpenID|Vector|WikiEditor|Collection)> allow from all </Directory> RedirectMatch ^/forum/?$ http://en.qi-hardware.com/wiki/Talk:Main_Page RedirectMatch ^/webchat/?$ http://webchat.freenode.net/?randomnick=1\&channels=qi-hardware RedirectMatch ^/feed/atom.xml$ http://en.qi-hardware.com/w/index.php?title=News\&action=feed\&feed=atom RedirectMatch ^/feed/rss20.xml$ http://en.qi-hardware.com/w/index.php?title=News\&action=feed\&feed=rss </VirtualHost>
~# a2ensite en.qi-hardware.com
~# touch /var/log/wiki-dump.log ~# chown www-data.www-data /var/log/wiki-dump.log ~# vi /etc/cron.d/wiki-dump 16 3 * * mon www-data php /srv/www/en/w/maintenance/dumpBackup.php --full --uploads --output=bzip2:/srv/www/downloads.qi-hardware.com/qi-wiki-dump.xml.bz2.in_progress --quiet >> /var/log/wiki-dump.log && mv /srv/www/downloads.qi-hardware.com/qi-wiki-dump.xml.bz2.in_progress /srv/www/downloads.qi-hardware.com/qi-wiki-dump.xml.bz2
eggdrop
~# apt-get install eggdrop ~# adduser --system eggdrop ~# mkdir /home/eggdrop/eggdrop-logs /home/eggdrop/qi-hardware-logs ~# chown eggdrop.nogroup /home/eggdrop/eggdrop-logs /home/eggdrop/qi-hardware-logs ~# zcat /usr/share/doc/eggdrop-data/examples/eggdrop.conf.gz > /home/eggdrop/eggdrop.conf ~# chmod a+x /home/eggdrop/eggdrop.conf
~# vi /home/eggdrop/eggdrop.conf (0755)
set username "qi-bot"
set admin "Admin <email: webmaster@qi-hardware.com>"
set network "irc.freenode.net"
set timezone "UTC"
set offset "0"
set env(TZ) "$timezone $offset"
set quick-logs 1
logfile mco * "eggdrop-logs/eggdrop"
logfile p #qi-hardware "qi-hardware-logs/qi-hardware"
set keep-all-logs 1
set logfile-suffix "_%Y%m%d.log"
set switch-logfiles-at 000
set quiet-save 1
set userfile "eggdrop.user"
set pidfile "pid.eggdrop"
listen 3852 all @localhost
set owner "wolfspraul"
#set whois-fields "url birthday"
#die "Please make sure you edit your config file completely."
set chanfile "eggdrop.chan"
channel add #qi-hardware {
chanmode "+nt-likm"
}
set net-type 5
set nick "qi-bot"
set altnick "qi-bot?"
set realname "qi eggdrop bot"
set identpass "_password_"
bind evnt - init-server evnt:init_server
proc evnt:init_server {type} {
global botnick identpass
putquick "MODE $botnick +i-ws"
putquick "PRIVMSG nickserv :identify $identpass"
}
bind notc - "*This nickname is registered. Please choose a different nickname,*" identify:notc
proc identify:notc { nick uhost handle text dest } {
global botnick identpass
if { $nick == "NickServ" } {
puthelp "PRIVMSG nickserv :identify $identpass"
putlog "Identifying as $botnick to $nick"
}
}
set servers {
irc.freenode.net
}
set flood-msg 0:0
set flood-ctcp 0:0
set notefile "eggdrop.notes"
#die "You didn't edit your config file completely like you were told, did you?"
source /home/eggdrop/scripts/socket-stuff.tcl
~# vi /home/eggdrop/scripts/socket-stuff.tcl # source of this script: http://paste.tclhelp.net/?id=7el # another script that might be handy one day: Selflog http://thommey.tclhelp.net/?page=scripts # good infos about special characters: http://www.peterre.info/characters.html set openport [listen 3858 script open:connection] proc open:connection { newidx } { putlog "(DEBUG) New connection from [findidxhost $newidx] ($newidx)" if {"[findidxhost $newidx]" != "telnet@localhost" && "[findidxhost $newidx]" != "telnet@fidelio.qi-hardware.com"} {killdcc $newidx} putidx $newidx "CONNECTED" control $newidx check:connection } proc check:connection {idx var} { set command [string toupper [lindex [split $var] 0]] switch $command { SAY { set channel [join [lindex [split $var] 1]] set msg [join [lrange [split $var] 2 end]] putserv "PRIVMSG $channel :$msg" putloglev p $channel "<qi-bot> $msg" putidx $idx "YO" } CLOSE { putlog "(DEBUG) Closing connection from [findidxhost $idx] ($idx)" killdcc $idx } PING { putidx $idx "PONG [unixtime]" putlog "(DEBUG) Ping request from [findidxhost $idx] ($idx)" } default { putidx $idx "ERROR Unknown command received ($var)" putlog "(DEBUG) ERROR: '$var' ([findidxhost $idx] ($idx))" } } } proc findidxhost {idx} { foreach c [dcclist] { if {[lindex $c 0] == $idx} { return "[lindex $c 2]" } } return 0 }
- for first-time initialization, run eggdrop with -m and identify yourself as owner to set password
~# cd /home/eggdrop ~# sudo -u eggdrop ./eggdrop.conf -m
~# vi /etc/init.d/eggdrop (0755)
#!/bin/sh
EGGDROP="/usr/bin/eggdrop"
OPTIONS="/home/eggdrop/eggdrop.conf"
# Check for daemon presence
[ -x "$EGGDROP" ] || exit 0
# Get lsb functions
. /lib/lsb/init-functions
. /etc/default/rcS
case "$1" in
start)
log_begin_msg "Starting eggdrop..."
start-stop-daemon --start --quiet --oknodo --chuid eggdrop --chdir /home/eggdrop --pidfile /home/eggdrop/pid.eggdrop --exec "$EGGDROP" -- $OPTIONS
log_end_msg $?
;;
stop)
log_begin_msg "Stopping eggdrop..."
start-stop-daemon --stop --pidfile /home/eggdrop/pid.eggdrop
log_end_msg $?
;;
restart|reload|force-reload)
$0 stop
sleep 1
$0 start
;;
status)
status_of_proc "$EGGDROP" eggdrop
;;
*)
log_success_msg "Usage: /etc/init.d/eggdrop {start|stop|restart|reload|force-reload|status}"
exit 1
esac
~# update-rc.d eggdrop defaults ~# invoke-rc.d eggdrop start
~# apt-get install libexpect-php5 expect
~# vi /srv/irc/send_irc_lines.php (0755)
<?php
ini_set("expect.timeout", 3);
ini_set("expect.loguser", 0);
# log file may create permission problems when run from web server
# ini_set("expect.logfile", "/tmp/eggdrop.telnet");
$stream = expect_popen("telnet localhost 3858");
$ret = expect_expectl($stream, array(array ("CONNECTED", 'CONNECTED', EXP_EXACT)));
if ($ret != 'CONNECTED') {
fprintf(STDOUT, "Error: Did not see CONNECTED ($ret).\n");
goto out;
}
while ($l=rtrim(fgets(STDIN), '\n')) {
if (strlen($l) > 0) {
fwrite ($stream, "say #qi-hardware $l\n");
$ret = expect_expectl($stream, array(array ("YO", 'YO', EXP_EXACT)));
if ($ret != 'YO') {
fprintf(STDOUT, "Error: Did not get command feedback ($ret).\n");
goto out;
}
}
}
out:
fclose ($stream);
?>
~# telnet localhost 3852 * registering qi-bot with freenode .msg nickserv register _qi-bot-password_ webmaster@qi-hardware.com --- receive the confirmation email --- .msg nickserv verify register qi-bot _secret_key_from_mail_ .msg nickserv set enforce on * to switch the current console channel: .console #qi-hardware * to update the topic of the current console channel: .topic hello world * if a nick gets juped, release with .msg nickserv release qi-bot _qi-bot-password_ * if a sender gets ignored, see the list of ignores with .ignores, remove someone with .-ignore 1 * to send to a channel: .say #qi-hardware hello world * to set the founder of a channel: .msg chanserv set #qi-hardware founder qi-bot * to see the flags of a channel: .msg chanserv flags #qi-hardware
qi-bot is making use of third-party gseen and stats modules: http://www.kreativrauschen.com/eggdrop.php These modules are very popular and commonly used for eggdrop. However, they are not packaged in debian and had to be built from source.
TODO: should the build process be described?
After the build process, place gseen.so and stats.so to /usr/lib/eggdrop/modules/.
changes in eggdrop.conf: source /home/eggdrop/stats.conf source /home/eggdrop/gseen.conf
new files in /home/eggdrop: stats.conf stats.lang gseen.conf gseen.en.lang
stats.lang and gseen.en.lang are stock files provided by stats and gseen modules.
stats.conf and gseen.conf are modified in comparison to stock files provided by stats and gseen modules:
$1File:stats.conf]] $1File:gseen.conf]]
Public commands:
*!top10 [ordering] lists the top10 users in the channel *!top20 [ordering] same as !top10, but lists the top20 (surprise!) *!stat [user] shows the statistics for the user *!place [user/ordering] shows on which place the user is *!ttop10/!ttop20/!tstat/!tplace same as above, but only uses the statistics of today
*!wordstats [user] lists the most used words of the user *!topwords lists the most used words in the channel *!top10 word <word> lists the top10 people who used <word>
*!seen <nick> I think this command doesn't need an explanation. ^_^ *!seen <mask> Searches the database for entries that match <mask> for example "!seen *!user@dialin-*.isp.com" *!seennick <nick> !seen also checks if a user was online later with a different nick. !seennick only seens for <nick> *!seenstats just a little report on how many nicks are tracked
irclog2html
- homepage http://mg.pov.lt/irclog2html/ https://launchpad.net/irclog2html
-
bzr branch lp:irclog2html, backed up in /srv/install-archive/irclog2html-bzr20110120.tar.bz2
~# tar -C /srv/irc -xjf /srv/install-archive/irclog2html-bzr20110120.tar.bz2
~# vi /etc/cron.d/irclog2html 11,41 * * * * eggdrop python /srv/irc/irclog2html/src/irclog2html/logs2html.py --title "#qi-hardware IRC Logs" --prefix "#qi-hardware IRC log for " --searchbox /home/eggdrop/qi-hardware-logs && python /srv/irc/irclog2html/src/irclog2html/logs2html.py --title "#milkymist IRC Logs" --prefix "#milkymist IRC log for " --searchbox /home/eggdrop/milkymist-logs && python /srv/irc/irclog2html/src/irclog2html/logs2html.py --title "#homecmos IRC Logs" --prefix "#homecmos IRC log for " --searchbox /home/eggdrop/homecmos-logs
~# vi /etc/apache2/sites-available/en.qi-hardware.com RewriteRule ^/irclogs/search/$ /irclogs/search [R,L] ScriptAlias /irclogs/search /srv/irc/irclog2html/src/irclog2html/irclogsearch.py <Location /irclogs/search> SetEnv IRCLOG_LOCATION /home/eggdrop/qi-hardware-logs/ </Location> Alias /irclogs /home/eggdrop/qi-hardware-logs <Directory /home/eggdrop/qi-hardware-logs> DirectoryIndex index.html Options FollowSymLinks AllowOverride None </Directory> Alias /mmlogs/irclog.css /srv/irc/irclog2html/src/irclog2html/irclog.css Alias /mmlogs/listCollapse.js /srv/irc/irclog2html/src/irclog2html/listCollapse.js ScriptAlias /mmlogs/search /srv/irc/irclog2html/src/irclog2html/irclogsearch.py <Location /mmlogs/search> SetEnv IRCLOG_LOCATION /home/eggdrop/milkymist-logs/ </Location> Alias /mmlogs /home/eggdrop/milkymist-logs <Directory /home/eggdrop/milkymist-logs> DirectoryIndex index.html Options FollowSymLinks AllowOverride None </Directory> Alias /homecmos-logs/irclog.css /srv/irc/irclog2html/src/irclog2html/irclog.css Alias /homecmos-logs/listCollapse.js /srv/irc/irclog2html/src/irclog2html/listCollapse.js ScriptAlias /homecmos-logs/search /srv/irc/irclog2html/src/irclog2html/irclogsearch.py <Location /homecmos-logs/search> SetEnv IRCLOG_LOCATION /home/eggdrop/homecmos-logs/ </Location> Alias /homecmos-logs /home/eggdrop/homecmos-logs <Directory /home/eggdrop/homecmos-logs> DirectoryIndex index.html Options FollowSymLinks AllowOverride None </Directory>
The irclog2html source has been patched to enable expanding/collapsing of logs grouped by year/month and stricter xhtml conformance.
The diff is here: http://bazaar.launchpad.net/~bas-bmail/irclog2html/grouped-list/revision/92?start_revid=92
InDefero
~# apt-get install php-pear php-mail php-mail-mime php-net-smtp
- 1) update
~# mkdir /home/www ~# chown www-data.www-data /home/www ~# cd /home/www ~# mysqldump indefero > indefero-yyyymmdd.sql
- 1.1) stop site
~# a2dissite projects.qi-hardware.com ~# /etc/init.d/apache2 reload ~# rm /tmp/*.phps ~# rm -r /tmp/cache
- 1.2) pluf
~# wget -O pluf-yyyymmdd.zip http://projects.ceondo.com/p/pluf/source/download/master/ ~# unzip pluf-yyyymmdd.zip ~# patch -p0 < --- File.php +++ /home/www/pluf-master/src/Pluf/Form/Field/File.php @@ -25,7 +25,7 @@ - public $max_size = 2097152; // 2 MB + public $max_size = 200000000; // 200 MB ~# chown -R www-data.www-data pluf-master ~# mv pluf pluf-yyyymmdd.archive ~# mv pluf-master pluf
- 1.3) Indefero
~# wget -O indefero-yyyymmdd.zip http://projects.ceondo.com/p/indefero/source/download/master/ ~# unzip indefero-yyyymmdd.zip ~# patch -p0 < --- Upload.php +++ /home/www/indefero-master/src/IDF/Form/Upload.php @@ -58,7 +58,7 @@ - 'max_size' => Pluf::f('max_upload_size', 2097152), + 'max_size' => Pluf::f('max_upload_size', 200000000), ~# cp -a indefero/attachments indefero-master ~# cp -a indefero/www/media/upload indefero-master/www/media ~# cp indefero-master/src/IDF/conf/path.php-dist indefero-master/src/IDF/conf/path.php
- 1.4) templates
~# mkdir -p indefero-master/customtemplates/idf
* indefero/master/customtemplates/idf/{qi-header.html,qi-footer.html} based on /srv/www/downloads.qi-hardware.com/{HEADER.shtml,FOOTER.shtml}
~# cp indefero-master/src/IDF/templates/idf/{base.html,base-simple.html,base-full.html} indefero-master/customtemplates/idf
* in all three templates, change as follows
{block extraheader}{/block}
+<link rel="stylesheet" type="text/css" href="http://downloads.qi-hardware.com/css/style.css" />
...
<body>
+{include 'idf/qi-header.html'}
...
+{include 'idf/qi-footer.html'}
</body>
- 1.5) config file
~# cp indefero-master/src/IDF/conf/idf.php-dist indefero-master/src/IDF/conf/idf.php * edit indefero-master/src/IDF/conf/idf.php $cfg['debug'] = false; $cfg['git_remote_url'] = 'git://projects.qi-hardware.com/%s.git'; $cfg['git_write_remote_url'] = 'git@projects.qi-hardware.com:%s.git'; $cfg['idf_plugin_syncgit_path_gitserve'] = '/home/www/indefero/scripts/gitserve.py'; # yes .py $cfg['idf_plugin_syncgit_path_authorized_keys'] = '/home/git/.ssh/authorized_keys'; $cfg['idf_plugin_syncgit_sync_file'] = '/tmp/SYNC-GIT'; $cfg['idf_plugin_syncgit_remove_orphans'] = false; $cfg['idf_plugin_syncgit_git_home_dir'] = '/home/git'; $cfg['idf_plugin_syncgit_base_repositories'] = '/home/git/repositories'; $cfg['svn_remote_url'] = 'http://projects.qi-hardware.com/svn/%s/trunk'; $cfg['idf_plugin_syncsvn_authz_file'] = '/home/svn/dav_svn.authz'; $cfg['idf_plugin_syncsvn_passwd_file'] = '/home/svn/dav_svn.passwd'; $cfg['idf_plugin_syncsvn_svn_path'] = '/home/svn/repositories'; $cfg['idf_plugin_syncsvn_remove_orphans'] = false; $cfg['admins'] = array(array('Admin', 'webmaster@qi-hardware.com')); $cfg['url_base'] = 'http://projects.qi-hardware.com'; $cfg['url_media'] = 'http://projects.qi-hardware.com/media'; $cfg['url_upload'] = 'http://projects.qi-hardware.com/media/upload'; $cfg['secret_key'] = '_secret_'; $cfg['from_email'] = 'discussion@lists.en.qi-hardware.com'; $cfg['bounce_email'] = 'no-reply@qi-hardware.com'; $cfg['db_login'] = 'drupal'; $cfg['db_password'] = '_password_'; $cfg['db_server'] = 'localhost'; $cfg['db_version'] = '5.0'; # Only needed for MySQL $cfg['db_engine'] = 'MySQL'; # SQLite is also well tested or MySQL $cfg['db_database'] = 'indefero'; # put absolute path to the db if you $cfg['idf_extra_upload_ext'] = ' deb cfg'; $cfg['max_upload_size'] = 200000000; // Size in bytes $cfg['time_zone'] = 'Europe/Berlin'; $cfg['template_folders'] = array('/home/www/indefero/customtemplates', dirname(__FILE__).'/../templates' ); $cfg['allowed_scm'] = array('git' => 'IDF_Scm_Git', 'svn' => 'IDF_Scm_Svn' );
- 1.6) migration
~# chown -R www-data.www-data /home/www/indefero-master ~# mv indefero indefero-yyyymmdd.archive ~# mv indefero-master indefero ~# cd /home/www/indefero/src ~# php /home/www/pluf/src/migrate.php --conf=IDF/conf/idf.php -a -d -u (to test) ~# php /home/www/pluf/src/migrate.php --conf=IDF/conf/idf.php -a -d (for real upgrade)
- 1.7) start site
~# a2ensite projects.qi-hardware.com ~# /etc/init.d/apache2 reload
- git - follow /home/www/indefero/doc/syncgit.mdtext
apt-get install git-core *) git-daemon-run not needed, indefero uses it’s own config
~# adduser --system --shell /bin/sh --gecos 'git version control' \ --group --disabled-password --home /home/git git
~# su git ~# mkdir -m 0700 /home/git/.ssh ~# touch /home/git/.ssh/authorized_keys ~# chmod 0600 /home/git/.ssh/authorized_keys ~# mkdir /home/git/repositories ~# usermod -a -G git www-data ~# echo "* * * * * php /home/www/indefero/scripts/gitcron.php" | crontab -u git -
~# apt-get install inetutils-inetd ~# vi /etc/inetd.d/git-daemon git stream tcp nowait git /usr/bin/git git daemon --inetd --base-path=/home/git/repositories ~# /etc/init.d/inetutils-inetd restart
- /home/www/indefero/doc/syncsvn.mdtext
~# apt-get install subversion libapache2-svn
~# vi /etc/apache2/sites-available/projects.qi-hardware.com <Location /svn> DAV svn SVNParentPath /home/svn/repositories AuthzSVNAccessFile /home/svn/dav_svn.authz Satisfy Any Require valid-user AuthType Basic AuthName "Subversion Repository" AuthUserFile /home/svn/dav_svn.passwd </Location>
~# mkdir -p /home/svn/repositories ~# touch /home/svn/dav_svn.authz ~# touch /home/svn/dav_svn.passwd ~# chown -R www-data:www-data /home/svn
~# pear install File_Passwd
~# mkdir /srv/www/projects.qi-hardware.com ~# chown www-data.www-data /srv/www/projects.qi-hardware.com ~# ln -s /home/www/indefero/www/media /srv/www/projects.qi-hardware.com ~# chown -h www-data.www-data /srv/www/projects.qi-hardware.com/media ~# ln -s /home/www/indefero/www/index.php /srv/www/projects.qi-hardware.com ~# chown -h www-data.www-data /srv/www/projects.qi-hardware.com/index.php
~# vi /srv/www/projects.qi-hardware.com/.htaccess
Options +FollowSymLinks
RewriteEngine On
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^(.*) /index.php/$1
~# vi /etc/apache2/sites-available/projects.qi-hardware.com <VirtualHost *:80> ServerName projects.qi-hardware.com DocumentRoot /srv/www/projects.qi-hardware.com <Directory /srv/www/projects.qi-hardware.com> Options Indexes -MultiViews +SymLinksIfOwnerMatch AllowOverride All Order allow,deny allow from all AddType application/x-httpd-php .php DirectoryIndex index.html index.php php_value upload_max_filesize 100M php_value post_max_size 200M </Directory> Alias /schhist /mnt/fidelio-schhist-output <Directory /mnt/fidelio-schhist-output> Order Deny,Allow Allow from all Options None AllowOverride None AddDefaultCharset utf-8 </Directory> </VirtualHost>
~# a2enmod rewrite.log ~# a2ensite projects.qi-hardware.com
- short domain qi-hw.com for IRC logging
~# vi /etc/apache2/sites-available/en.qi-hardware.com <VirtualHost *:80> ServerName qi-hw.com RedirectMatch permanent ^/p/(.*)/(.*)$ http://projects.qi-hardware.com/index.php/p/$1/source/commit/$2 RedirectMatch permanent .* http://en.qi-hardware.com/wiki/Main_Page </VirtualHost>
~# vi /etc/ssh/sshd_config AllowUsers ... git
*) anonymous (read-only) access: git clone git://projects.qi-hardware.com/board-qi-avt2.git
*) authenticated (read-write) access: git clone git@projects.qi-hardware.com:board-qi-avt2.git
*) set or reset admin flag for a user mysql indefero -e "update indefero_users set administrator=1/0 where login = 'user_login';"
git
- after creating a new project in Indefero, the git repository is in /home/git/repositories/[short-name], but is only initialized after the first push.
- the ./description is not setup automatically by Indefero:
~# vi /home/git/repositories/[short-name]/description short-name
- /srv/irc/post-receive-email-irc
~# cp /usr/share/doc/git-core/contrib/hooks/post-receive-email /srv/irc/post-receive-email-irc
~# vi /srv/install-archive/post-receive-hack.patch
--- /usr/share/doc/git-core/contrib/hooks/post-receive-email 2009-11-21 09:47:52.000000000 -0500
+++ post-receive-email-irc 2010-04-27 23:53:32.000000000 -0400
@@ -123,6 +123,8 @@
# branch
refname_type="branch"
short_refname=${refname##refs/heads/}
+# Qi: IRC logging (don't disturb sendmail, so STDOUT redirected to STDERR)
+git log -20 $oldrev..$newrev --reverse --pretty=format:"%an: %s ($short_refname) http://qi-hw.com/p/$projectdesc/%h" | php /srv/irc/send_irc_lines.php 1>&2
;;
@@ -150,8 +152,9 @@
- echo >&2 "*** $config_name is not set so no email will be sent"
- echo >&2 "*** for $refname update $oldrev->$newrev"
+# Qi: we call this script also for just the IRC logging, so not stderr output wanted
+# echo >&2 "*** $config_name is not set so no email will be sent"
+# echo >&2 "*** for $refname update $oldrev->$newrev"
@@ -589,6 +592,10 @@
send_mail()
{
+# Qi: make send_mail() fail gracefully for IRC logging only
+ if [ -z "$recipients" ]; then
+ exit 0
+ fi
~# cd /srv/irc && patch -p0 < /srv/install-archive/post-receive-hack.patch
- always execute the post-receive-email-irc script, it will fail gracefully if no commitlog mailing list is setup, and always notify the IRC channel
~# vi /home/git/repositories/[short-name]/hooks/post-receive (0755) . /srv/irc/post-receive-email-irc
- commitlog mailing list (currently only openwrt-xburst, openwrt-packages and xburst-tools):
~# cd /home/git/repositories/[short-name] *) You can edit the ./config file manually or via git-config ~# git config hooks.mailinglist "commitlog@lists.en.qi-hardware.com" *) note that only one mailinglist is supported by the post-receive-email script right now, to support more we might have to add a --get-all parameter into the script ~# git config hooks.emailprefix "[short-name] " ~# git config hooks.envelopesender developer@qi-hardware.com
- schhist (one time)
~# ssh-keygen -f /home/git/.ssh/id_rsa -C git@turandot -N ""
~# chown git:git /home/git/.ssh/{id_rsa,id_rsa.pub}
*) copy /home/git/.ssh/id_rsa.pub to fidelio.qi-hardware.com:/home/schhist/.ssh/authorized_keys
- schhist (for each project with KiCad files)
~# vi /home/git/repositories/[short-name]/hooks/post-receive (0755) ssh schhist@fidelio.qi-hardware.com touch git/_short-name_.pull_needed
Piwik
- homepage http://piwik.org
- download v1.0 from http://piwik.org/latest.zip and save in /srv/install-archive/piwik_1_0_20100901.zip
~# mkdir /srv/www/en/piwik
~# unzip /srv/install-archive/piwik_1_0_20100901.zip -d /srv/www/en/piwik
~# rm /srv/www/en/piwik/{Manifest.xml,parameters.xml,How\ to\ install\ Piwik.html}
~# chmod 0755 /srv/www/en/piwik/misc/cron/archive.sh
~# mysql e 'grant all on piwik_db.* to 'drupal'@'localhost';'
- go to http://en.qi-hardware.com/piwik and follow setup process
Database Setup login: drupal database name: piwik_db General Setup super user login: xxx password: xxx email: mailman@lists.en.qi-hardware.com disabled community updates Setup a Website website name: Qi Hardware website url: http://en.qi-hardware.com website time zone: UTC Settings, General Settings Allow Piwik archiving to trigger when reports are viewed from the browser: No Reports for today will be processed at most every: 3600 seconds Settings, Users anonymous: View
~# vi /etc/apache2/sites-available/en.qi-hardware.com <Directory /srv/www/en/piwik> AllowOverride All </Directory>
~# vi /etc/php5/cli/conf.d/piwik.ini ; increase memory limit for the archive.sh script memory_limit = 192M
~# vi /etc/cron.d/piwik 22 * * * * www-data /srv/www/en/piwik/misc/cron/archive.sh > /dev/null
- update automatically via web interface
- ImageGraph and VisitsSummaryEvolution plugins for server-side generated PNG reports
- ImageGraph homepage http://dev.piwik.org/trac/ticket/1721
- VisitsSummaryEvolution homepage http://dev.piwik.org/trac/ticket/1773
~# wget -P /srv/install-archive http://dev.piwik.org/trac/raw-attachment/ticket/1721/ImageGraph.2.zip ~# unzip -d /srv/www/en /srv/install-archive/ImageGraph.2.zip ~# wget -P /srv/install-archive http://dev.piwik.org/trac/raw-attachment/ticket/1773/VisitsSummaryEvolution.zip ~# unzip -d /srv/www/en /srv/install-archive/VisitsSummaryEvolution.zip
~# cd /srv/www/en/piwik/plugins && patch -p0 <
+++ ImageGraph/API.php 2010-10-21 00:27:10.000000000 -0400
@@ -42,8 +42,8 @@
- Piwik::checkUserIsNotAnonymous();
- Piwik::checkUserHasViewAccess($idSite);
+// Piwik::checkUserIsNotAnonymous();
+// Piwik::checkUserHasViewAccess($idSite);
+++ ImageGraph/ImageGraphObject.php 2010-10-20 23:04:13.000000000 -0400
@@ -78,7 +78,7 @@
$this->drawGraphArea(255, 255, 255);
- $this->drawScale($this->data->GetData(), $this->data->GetDataDescription(), SCALE_START0, 68, 68, 68, true, 0, 2, false);
+ $this->drawScale($this->data->GetData(), $this->data->GetDataDescription(), SCALE_START0, 68, 68, 68, true, 0, 0, false); // set decimals to 0
$this->drawGrid(4, true, 230, 230, 230, 255);
+++ VisitsSummaryEvolution/API.php 2010-10-21 00:14:12.000000000 -0400
@@ -50,8 +50,8 @@
- Piwik::checkUserIsNotAnonymous();
- Piwik::checkUserHasViewAccess($idSite);
+// Piwik::checkUserIsNotAnonymous();
+// Piwik::checkUserHasViewAccess($idSite);
@@ -78,8 +78,8 @@
else if($period == "month")
{
- $dateHistory = $dateHistory->setDay(1);
- $dateTemp = Piwik_Date::factory($date)->addPeriod(1, "month")->setDay(1)->subDay(1);
+ $dateTemp = Piwik_Date::factory("yesterday");
+ $dateHistory = Piwik_Date::factory($dateTemp)->subPeriod(1, "month");
}
@@ -98,7 +98,7 @@
$temp = Piwik_VisitsSummary_API::getInstance()->get($idSite, $evolutionPeriod, $dateHistory)->getFirstRow()->getColumns();
- $reportTemp["nb_visits"] = $temp["nb_visits"];
+ $reportTemp["nb_visits"] = $temp["nb_uniq_visitors"];
$reportTemp["nb_uniq_visitors"] = $temp["nb_visits"];
$reportTemp["nb_visits_converted"] = $temp["nb_visits_converted"];
- go to http://en.qi-hardware.com/piwik, then
- Settings, Plugins, ImageGraph -> Activate
- Settings, Plugins, VisitsSummaryEvolution -> Activate
- setting &ordinateColumn=nb_uniq_visitors won't work because piwik/plugins/API/API.php:handleTableReport() will remove nb_uniq_visitors
pertain.qi-hardware.com
system
~# adduser --disabled-password <user> ~# vi /etc/hosts 127.0.0.1 pertain.qi-hardware.com pertain
locale
~# echo "en_US.UTF-8 UTF-8" >> /etc/locale.gen ~# apt-get -y --force-yes install locales ~# dpkg-reconfigure locales *) in dpkg-reconfigure, set default system locale to 'en_US.UTF-8'
packages
~# vi /etc/apt/sources.list deb http://ftp.debian.org/debian squeeze main deb-src http://ftp.debian.org/debian squeeze main deb http://ftp.debian.org/debian unstable main
~# vi /etc/apt/apt.conf.d/50local APT::Default-Release "squeeze"; APT::Get::Show-Upgraded "true"; APT::Get::Show-Versions "true";
~# apt-get install apt-file ~# apt-file update
- squeeze updates
~# apt-get update ~# apt-get upgrade
nfs
~# apt-get install nfs-kernel-server ~# vi /etc/exports /home/schhist/output turandot.qi-hardware.com(ro,async,all_squash,no_subtree_check) ~# /etc/init.d/nfs-kernel-server restart
~# apt-get install quilt devscripts strace bzr subversion git
httpd with userdir
~# apt-get install apache2 apache2-doc apache2-mpm-prefork apache2.2-common apache2-utils ~# a2enmod userdir
mysql
~# apt-get install mysql-server ~# mysqladmin -u root password _pwd_ ~# vi /root/.my.cnf [0600] [client] password=_pwd_
~# vi /etc/mysql/conf.d/qi.cnf # used mysqltuner and /etc/init.d/mysql status to optimize values [mysqld] table_cache = 200 thread_concurrency = 4 query_cache_limit = 32M query_cache_size = 128M join_buffer_size = 1M log_slow_queries = /var/log/mysql/mysql-slow.log long_query_time = 2 tmp_table_size = 192M max_heap_table_size = 192M innodb_buffer_pool_size = 256M wait_timeout = 60
- flush and lock databases for lvm snapshot
~# mysql -e 'flush tables with read lock; flush logs;'
- unlock databases after lvm snapshot is taken
~# mysql -e 'unlock tables;'
- move database to different server
src: mysqldump db_name > db_name.sql dest: mysql -e ‘create database db_name;’ dest: mysql db_name < db_name.sql *) Note that this did not work for Chinese characters in database fields, maybe some UTF-8 problem? *) --all-databases dumps all dbs, --xml dumps in xml format
- create new user
~# mysql -e 'create user 'drupal'@'localhost' identified by '_pwd_';' ~# mysql -e 'grant all on _db_name_.* to 'drupal'@'localhost';'
- change user password
~# mysql -e 'set password for 'drupal'@'localhost' = PASSWORD('_new_password_');'
- overwrite lost root password
~# /etc/init.d/mysql stop
~# /usr/bin/mysqld_safe --skip-grant-tables &
~# mysql --user=root mysql
mysql> update user set Password=PASSWORD('new-password-here') WHERE User='root';
mysql> flush privileges;
mysql> exit
~# fg
~# ctrl-c
~# /etc/init.d/mysql start
vnc
* had to upgrade to udev-unstable to fix lxc problem (debian bug #651941) ~# apt-get -t unstable install udev
~# apt-get install xorg xdm tightvncserver icewm
~# su - schhist --command="vncserver"
enter password (can also be set with vncpasswd)
~# vi /etc/init.d/vncserver (0755)
#!/bin/sh
### BEGIN INIT INFO
# Provides: vncserver
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
case "$1" in
start)
echo -n "Starting vnc server: "
su - schhist --command="vncserver :0 -geometry 1024x768 -depth 16"
;;
stop)
echo -n "Stopping vnc server: "
su - schhist --command="vncserver -kill :0"
;;
*)
echo "Usage: /etc/init.d/vncserver {start|stop}"
exit 1
;;
esac
exit 0
~# update-rc.d vncserver defaults
~# invoke-rc.d vncserver start
- client access to the VNC server
- tunnel over ssh:
ssh your_account@pertain.qi-hardware.com -L 5900:127.0.0.1:5900, thenvncviewer :0
kicad
~# apt-get build-dep kicad ~# bzr branch -r 3493 lp:kicad ~# git clone git://projects.qi-hardware.com/eda-tools.git ~# cd kicad ~# ln -s ../eda-tools/kicad-patches patches ~# quilt push -a ~# cmake -DKICAD_TESTING_VERSION=ON . ~# make ~# make install
- to try kicad, run
export DISPLAY=:0first as kicad will segfault without it
- Debian package (currently not used)
- find matching revisions for kicad-library and kicad-doc
- https://code.launchpad.net/~kicad-testing-committers/kicad/testing
- https://code.launchpad.net/~kicad-developers/kicad/doc
- https://code.launchpad.net/~kicad-lib-committers/kicad/library
~# bzr branch -r 2448 lp:kicad ~# bzr branch -r 93 lp:~kicad-lib-committers/kicad/library kicad-library ~# bzr branch -r 117 lp:~kicad-developers/kicad/doc kicad-doc
~# apt-get source --diff-only kicad
~# zcat kicad_0.0.20100314-1.diff.gz|patch -p1
~# patch -p0 <
+++ debian/rules 2010-10-19 14:24:12.430103046 +0000
@@ -64,7 +64,7 @@
cd $(CURDIR)/build/kicad/demos && cmake -P cmake_install.cmake
- cd $(CURDIR)/build/kicad/internat && cmake -DCMAKE_INSTALL_PREFIX=$(CURDIR)/debian/kicad-common/usr -P cmake_install.cmake
+ cd $(CURDIR)/build/kicad-doc/internat && cmake -DCMAKE_INSTALL_PREFIX=$(CURDIR)/debian/kicad-common/usr -P cmake_install.cmake
cd $(CURDIR)/build/kicad/template && cmake -DCMAKE_INSTALL_PREFIX=$(CURDIR)/debian/kicad-common/usr -P cmake_install.cmake
~# dch -l bzr2448.werner 'kicad bzr2448 plus werner' ~# debuild --no-lintian -us -uc ~# dpkg -i ../kicad_0.0.20100314-1bzr2448.werner1_amd64.deb ../kicad-common_0.0.20100314-1bzr2448.werner1_all.deb
fped
~# apt-get -t unstable install fped
schhist
~# adduser --disabled-password schhist ~# cd /home/schhist ~# git clone git://projects.qi-hardware.com/eda-tools.git ~# cd eda-tools/schhist/ppmdiff && make ~# mkdir output ~# cp eda-tools/schhist/demo-index.html output/index.html
~# vi /home/schhist/output/index.html
<HTML> <TITLE> Schematics Histories </TITLE> <BODY> <H1>Schematics Histories</H1> <A href="board-m1/">Milkymist One</A><BR> <A href="m1-gps-expansion/">M1 GPS expansion</A><BR> <A href="m1-jtag-serial/">Milkymist One JTAG-serial</A><BR> <A href="atben/">atben (ben-wpan)</A><BR> <A href="atusb/">atusb (ben-wpan)</A><BR> <A href="cntr/">cntr (ben-wpan)</A><BR> <A href="ben-blinkenlights/">Ben Blinkenlights</A><BR> <A href="f32xbase_c2ben/">c2ben (f32xbase)</A><BR> <A href="labsw/">labsw</A><BR> <A href="board-qi-avt2/">AVT2</A><BR> </BODY> </HTML>
~# vi /etc/cron.d/schhist 13 4 * * * schhist touch /home/schhist/git/pull_all * * * * * schhist /home/schhist/eda-tools/schhist/schhist-qi-update-all >>/home/schhist/cron.log 2>&1
fidelio.qi-hardware.com
This server is not backed up, see it like a big /tmp folder. Do not store original data on it. We use it as build host, but also to experiment with testing or unstable packages and as a sandbox to run potentially unsafe scripts for other servers. Uptime requirements are lower, so for example we are not running in a virtual machine. If updates make the machine unbootable, we have to go back to the Hetzner rescue console which can be slow.
hardware
- hetzner.de, Root Server X2, EUR 29 / month
- AMD 64 3700+, 2 GB DDR400 RAM, 2 x 300 GB SATA (Software-RAID 0)
- IPv4 213.239.211.82
- IPv6 2a01:4f8:61:6322:: /64 (::1 /59 is gateway)
- Operating System: Rescue system
- reverse DNS to fidelio.qi-hardware.com
base system
- ssh into rescue
# installimage o/s list: Debian (official) Debian images: Debian50-lenny-64-minimal config editor SWRAIDLEVEL 0 HOSTNAME fidelio PART /boot ext3 512M PART swap swap 16G PART / ext3 10G PART /home ext3 all # reboot
accounts
- add authorized_keys to root, then disable root password
passwd --delete root usermod --lock root
- user accounts
adduser --disabled-password <user>
package management
~# vi /etc/apt/sources.list deb http://ftp.uni-bayreuth.de/linux/Debian/debian/ squeeze main deb-src http://ftp.uni-bayreuth.de/linux/Debian/debian/ squeeze main
~# vi /etc/apt/apt.conf.d/50local APT::Get::Show-Upgraded "true"; APT::Get::Show-Versions "true";
~# apt-get install apt-file ~# apt-file update
- distribution upgrade to squeeze
~# apt-get update ~# apt-get install linux-image-2.6-amd64 ~# reboot (booting from newer kernel first to avoid udev problems in dist-upgrade) ~# apt-get dist-upgrade (accept defaults, overwrite old config files) ~# /usr/sbin/upgrade-from-grub-legacy install to /dev/sda and /dev/sdb ~# rm /boot/grub/menu.lst ~# vi /etc/default/grub GRUB_TIMEOUT=0 ~# update-grub ~# reboot
- could look into automatic updates, cron-apt or unattended-upgrades
- simulate only, then send mail: /usr/bin/apt-get update && /usr/bin/apt-get -s -u upgrade
performance
~# vi /etc/default/cpufrequtils GOVERNOR="performance"
~# umount /home ~# mke2fs -t ext4 -O ^has_journal /dev/md3 ~# vi /etc/fstab /dev/md3 /home ext4 noatime,defaults,data=writeback 0 0 ~# mount /home
NFS
- client
~# apt-get install nfs-common portmap ~# mkdir /mnt/turandot-git-repos ~# vi /etc/fstab turandot.qi-hardware.com:/home/git/repositories /mnt/turandot-git-repos nfs ro 0 0
- server
~# apt-get install nfs-kernel-server ~# vi /etc/exports /home/schhist/output turandot.qi-hardware.com(ro,async,all_squash,no_subtree_check) ~# /etc/init.d/nfs-kernel-server restart
~# apt-get install quilt devscripts strace
httpd with userdir
~# apt-get install apache2 apache2-doc apache2-mpm-prefork apache2.2-common apache2-utils ~# a2enmod userdir
VNC
~# apt-get install xorg xdm tightvncserver icewm
~# su - schhist --command="vncserver"
enter password (can also be set with vncpasswd)
~# vi /etc/init.d/vncserver (0755)
#!/bin/sh
### BEGIN INIT INFO
# Provides: vncserver
# Required-Start:
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
### END INIT INFO
case "$1" in
start)
echo -n "Starting vnc server: "
su - schhist --command="vncserver :0 -geometry 1024x768 -depth 16"
;;
stop)
echo -n "Stopping vnc server: "
su - schhist --command="vncserver -kill :0"
;;
*)
echo "Usage: /etc/init.d/vncserver {start|stop}"
exit 1
;;
esac
exit 0
~# update-rc.d vncserver defaults
~# invoke-rc.d vncserver start
- client access to the VNC server
- direct VNC protocol:
vncviewer fidelio.qi-hardware.com:0 - tunnel over ssh:
ssh your_account@fidelio.qi-hardware.com -L 5900:127.0.0.1:5900, thenvncviewer :0
KiCad
- We are trying to pick a recent yet stable revision from the kicad testing branch. Werner will do this a few times a year.
- The last revision Werner chose was bzr 2448.
- We then find matching revisions in the kicad-doc and kicad-library subprojects by rough date.
- https://code.launchpad.net/~kicad-testing-committers/kicad/testing
- https://code.launchpad.net/~kicad-developers/kicad/doc
- https://code.launchpad.net/~kicad-lib-committers/kicad/library
- The matching revisions for bzr 2448 are doc: 117, library: 93
- 1. get upstream sources, library and documentation
~# bzr branch -r 2448 lp:kicad ~# bzr branch -r 93 lp:~kicad-lib-committers/kicad/library kicad-library ~# bzr branch -r 117 lp:~kicad-developers/kicad/doc kicad-doc
- 2. apply Debian patches
~# apt-get source --diff-only kicad
~# zcat kicad_0.0.20100314-1.diff.gz|patch -p1
~# patch -p0 <
+++ debian/rules 2010-10-19 14:24:12.430103046 +0000
@@ -64,7 +64,7 @@
cd $(CURDIR)/build/kicad/demos && cmake -P cmake_install.cmake
- cd $(CURDIR)/build/kicad/internat && cmake -DCMAKE_INSTALL_PREFIX=$(CURDIR)/debian/kicad-common/usr -P cmake_install.cmake
+ cd $(CURDIR)/build/kicad-doc/internat && cmake -DCMAKE_INSTALL_PREFIX=$(CURDIR)/debian/kicad-common/usr -P cmake_install.cmake
cd $(CURDIR)/build/kicad/template && cmake -DCMAKE_INSTALL_PREFIX=$(CURDIR)/debian/kicad-common/usr -P cmake_install.cmake
- 3. apply Werner's patches
~# git clone git://projects.qi-hardware.com/eda-tools.git ~# cd eda-tools ~# git reset --hard 41a65c ~# cd .. ~# ln -s ../eda-tools/kicad-patches kicad/patches ~# cd kicad ~# quilt push -a ~# cd ..
- 4. perform actual build and install
~# apt-get install devscripts ~# apt-get build-dep kicad ~# dch -l bzr2448.werner 'kicad bzr2448 plus werner' ~# debuild --no-lintian -us -uc ~# dpkg -i ../kicad_0.0.20100314-1bzr2448.werner1_amd64.deb ../kicad-common_0.0.20100314-1bzr2448.werner1_all.deb
fped
- packages at http://projects.qi-hardware.com/p/fped/downloads
~# git clone git://projects.qi-hardware.com/fped.git ~# cd fped ~# git checkout origin/debian ~# ./debian/rules get-orig-source ~# mv fped_0.0+r5986.orig.tar.gz .. ~# debuild --no-lintian -us -uc ~# dpkg -i ../fped_0.0+r5986-1_amd64.deb
schhist
~# adduser --disabled-password schhist ~# cd /home/schhist ~# git clone git://projects.qi-hardware.com/eda-tools.git ~# cd eda-tools/schhist/ppmdiff && make ~# mkdir output ~# cp eda-tools/schhist/demo-index.html output/index.html
- edit output/index.html to list all projects
~# vi /home/schhist/schhist_update_all (0755)
#!/bin/bash
#
# Only run one instance of this script.
#
if pidof -x $(basename $0) > /dev/null; then
for p in $(pidof -x $(basename $0)); do
if [ $p -ne $$ ]; then
echo "Script $0 is already running: exiting"
exit
fi
done
fi
[ ! -d git ] && mkdir git
#
# pull_all is set once a day in case we are missing a commit event
#
if [ -e git/pull_all ]; then
rm git/pull_all
pull_all=1
else
pull_all=0
fi
# eeschema segfaults if DISPLAY is not set
export DISPLAY=:0
#
# $1 url to git repo
# $2 local directory name
#
function is_git_updated
{
if [ ! -d git/$2 ]; then
rm -f git/$2.last git/$2.pull_needed
(cd git && git clone $1 $2)
elif [ $pull_all -gt 0 ] || [ -e git/$2.pull_needed ]; then
rm -f git/$2.pull_needed
(cd git/$2 && git pull)
fi
if [ -e git/$2.last ]; then
LAST_COMMIT=`cat git/$2.last`
fi
CURRENT_COMMIT=`cd git/$2 && git rev-parse @{0}`
if [ "$LAST_COMMIT" != "$CURRENT_COMMIT" ]; then
echo $CURRENT_COMMIT > git/$2.last
return 1
fi
return 0
}
#
# run_schhist2web parameters
# -n -f -S -c cache: will be passed to schhist2web
# --order order: will become SCHHIST_ORDER
# --title title: will become SCHHIST_TITLE
# $1 local git directory
# $2 root schematic
# $3 output directory
# $4 remote url home
# $5 remote url commit template
#
function run_schhist2web
{
unset SCHHIST_TITLE
unset SCHHIST_ORDER
while true; do
case "$1" in
-n | -f | -S) schhist2web_options="$schhist2web_options $1"
shift;;
-c) schhist2web_options="$schhist2web_options $1 $2"
shift 2;;
--order) export SCHHIST_ORDER=$2; shift 2;;
--title) export SCHHIST_TITLE=$2; shift 2;;
*) break;;
esac
done
export SCHHIST_HOME_URL=$4
export SCHHIST_COMMIT_TEMPLATE=$5
export SCHHIST_ADDLIBS=device
./eda-tools/schhist/schhist2web $schhist2web_options git/$1 $2 $3
}
#
# main
#
is_git_updated git://projects.qi-hardware.com/kicad-libs.git kicad-libs
is_git_updated git://projects.qi-hardware.com/ben-wpan.git ben-wpan || (
run_schhist2web -S --title ben-wpan/atusb --order atusb:USB:RF \
ben-wpan atusb/atusb.sch output/atusb \
http://projects.qi-hardware.com/p/ben-wpan/ \
"http://projects.qi-hardware.com/p/ben-wpan/source/commit/{}/"
run_schhist2web -S --title ben-wpan/cntr \
ben-wpan cntr/cntr.sch output/cntr \
http://projects.qi-hardware.com/p/ben-wpan/ \
"http://projects.qi-hardware.com/p/ben-wpan/source/commit/{}/"
run_schhist2web -S --title ben-wpan/atben \
ben-wpan atben/atben.sch output/atben \
http://projects.qi-hardware.com/p/ben-wpan/ \
"http://projects.qi-hardware.com/p/ben-wpan/source/commit/{}/"
)
is_git_updated https://github.com/milkymist/board-m1.git board-m1 || (
run_schhist2web -S --title "Milkymist One" --order m1:POWER:FPGA_Dec:BANK0:BANK1:BANK2:BANK3:EXPANSION_RESET:DRAM:NOR_FLASH:AUDIO:DVI-I:VIDEO_IN:MISC_CONTROL:ETHERNET:USB_A-B:USB_C-D:USB_E-F:MISC \
board-m1 r4/m1.sch output/board-m1 \
https://github.com/milkymist/board-m1/ \
"https://github.com/milkymist/board-m1/commit/{}"
)
is_git_updated https://github.com/kristianpaul/m1-gps-expansion.git m1-gps-expansion || (
run_schhist2web -S --title "M1 GPS expansion board" \
m1-gps-expansion gps.sch output/m1-gps-expansion \
https://github.com/kristianpaul/m1-gps-expansion/ \
"https://github.com/kristianpaul/m1-gps-expansion/commit/{}"
)
is_git_updated git://projects.qi-hardware.com/m1.git m1 || (
run_schhist2web -S --title "Milkymist One JTAG-serial" \
m1 jtag-serial/usb_jtag.sch output/m1-jtag-serial \
http://projects.qi-hardware.com/p/m1/ \
"http://projects.qi-hardware.com/p/m1/source/commit/{}/"
)
is_git_updated git://projects.qi-hardware.com/ben-blinkenlights.git ben-blinkenlights || (
run_schhist2web -S --title "Ben Blinkenlights" \
ben-blinkenlights bbl.sch output/ben-blinkenlights \
http://projects.qi-hardware.com/p/ben-blinkenlights/ \
"http://projects.qi-hardware.com/p/ben-blinkenlights/source/commit/{}/"
run_schhist2web -S --title "UBB VGA (ben-blinkenlights)" \
ben-blinkenlights ubb-vga/ubb-vga.sch output/ubb-vga \
http://projects.qi-hardware.com/p/ben-blinkenlights/ \
"http://projects.qi-hardware.com/p/ben-blinkenlights/source/commit/{}/"
)
is_git_updated git://projects.qi-hardware.com/board-qi-avt2.git board-qi-avt2 || (
run_schhist2web -S --title AVT2 \
board-qi-avt2 main/Qi_AVT2.sch output/board-qi-avt2 \
http://projects.qi-hardware.com/p/board-qi-avt2/ \
"http://projects.qi-hardware.com/p/board-qi-avt2/source/commit/{}/"
)
is_git_updated git://projects.qi-hardware.com/f32xbase.git f32xbase || (
run_schhist2web -S --title c2ben \
f32xbase c2ben/c2ben.sch output/f32xbase_c2ben \
http://projects.qi-hardware.com/p/f32xbase/ \
"http://projects.qi-hardware.com/p/f32xbase/source/commit/{}/"
)
is_git_updated git://projects.qi-hardware.com/wernermisc.git wernermisc || (
run_schhist2web -S --title labsw \
wernermisc labsw/labsw.sch output/labsw \
http://projects.qi-hardware.com/p/wernermisc/ \
"http://projects.qi-hardware.com/p/wernermisc/source/commit/{}/"
)
~# vi /etc/cron.d/schhist 13 4 * * * schhist touch /home/schhist/git/pull_all * * * * * schhist /home/schhist/schhist_update_all >>/home/schhist/cron.log 2>&1
- temporary KiCad bug workaround
schhist sometimes picks up ghost differences between revisions, and after investigation it looks like there might be an uninitialized variable in eeschema that makes line widths appear either at (randomly) 60 or 120 units (maybe deci-mil). Turning off the kernel's memory randomization helped, for now, in making eeschema stable enough to always draw the same line width. After a major KiCad upgrade we should check whether this is still needed, by looking for ghost diffs in large schematics histories such as the ones of AVT2 or Xué, for example.
~# vi /etc/sysctl.conf/kicad-eeschema-ghost-diffs-fix.conf kernel.randomize_va_space = 0
Recover From Snapshot
- this documents the rough procedure how to create a new server from a backed up snapshot tarball
- boot into Hetzner rescue system
~# parted /dev/sda mklabel msdos mkpart primary 2MB 500MB set 1 boot on mkpart primary 500MB 100% set 2 lvm on quit
- grub on Hetzner's rescue OS had trouble with ext4, so make the boot partition ext2 for now
~# mkfs.ext2 /dev/sda1 ~# pvcreate /dev/sda2 ~# vgcreate vg_mimi /dev/sda2
- set swap size to about 2 times RAM
~# lvcreate --size 1024M --name lv_swap vg_mimi
- leave 20% for snapshots
~# lvcreate --extents 80%FREE --name lv_root vg_mimi ~# mkswap /dev/vg_mimi/lv_swap ~# mkfs.ext4 /dev/vg_mimi/lv_root
- recover rootfs from tarball
~# mkdir /tmp/rootfs ~# mount /dev/vg_mimi/lv_root /tmp/rootfs ~# cd /mnt/rootfs ~# wget -O - http://.../-rootfs.tar.gz | tar -xz or ~# ssh user@backup-system cat .../-rootfs.tar.gz | tar -xz ~# mount /dev/sda1 boot
- all files in the boot tarball should start in the boot/ subdirectory
~# wget -O - http://.../-boot.tar.gz | tar -xz or ~# ssh user@backup-system cat .../-boot.tar.gz | tar -xz ~# grub-install --root-directory=/tmp/rootfs /dev/sda ~# umount boot
- .autorelabel will verify and set labels, otherwise ssh logins may fail. Perhaps because pty devices have an 'altered' selinux label after rescue. Also look at restorecon.
~# touch .autorelabel
- if ./etc/fstab uses UUID labels, set them correctly
~# vol_id /dev/sda1 ~# tune2fs /dev/sda1 -U _UUID_label_from_fstab_
- make sure the network is setup correctly, for example in /etc/sysconfig/network-scripts/ifcfg-eth0 or /etc/network/interfaces
~# cd .. ~# umount rootfs ~# reboot
