Wednesday, February 21, 2018

SKLearn Metrics Precision, Recall, F-Score and Support

Precision:
  • If it's 1 it means all positives are correct
    • It may be that many other positives for this class are wrongly labelled as negative (and categorised in other classes), but that's not measured in this metric
  • The lower it is, it means more entries from other categories are being mis-classified here
Recall:
  • If it's 1 it means it managed to find all the positive samples
    • It may be that it wrongly classified other samples for other categories here, but that's not measured in this metric
  • The lower it is, it means more entries from this category are being mis-classified into other categories
F-Score: Some kind of mean between precision and recall

Support: Number of occurrence in each category

Reference: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html

Thursday, October 12, 2017

Article About Business Rules Versus Machine Learning

We usually program logic using business rules. However, there are cases where business rules are not clear cut and need continuous iterations to get right. This can be costly and difficult. In these cases, the machine learning approach might help.

Article: https://www.linkedin.com/pulse/data-science-machine-learning-vs-rules-based-karthik-guruswamy/

Tuesday, February 14, 2017

Resume SCP Transfer

No way to do in SCP. Use rsync:
rsync --partial --progress --rsh=ssh ec2-user@xxx.xxx.xxx.xxx:abc.tgz abc.tgz

Reference: https://yyab.wordpress.com/2006/12/18/resume-a-large-scp-transfer/

Thursday, January 12, 2017

Tomcat Connections Getting Disconnected After Period of Inactivity

In some network environments, the firewall or router might terminate some connections after a period of inactivity.

However, this is not enabled by default:
# netstat -anpo | grep :80
tcp6       0      0 192.168.1.7:80      123.123.123.123:38123    ESTABLISHED 21345/java           off (0.00/0/0)

To prevent this, KeepAlive must be enabled in Tomcat. This is done by editing the Connector configuration in the server.xml file and adding the "socket.soKeepAlive" attribute.
   <Connector port="8080" protocol="HTTP/1.1"
              connectionTimeout="20000"
              redirectPort="8443"
               socket.soKeepAlive="true"
        />

However, this alone may not be enough, because by default in some Linux distros, the default KeepAlive time is 2 hours, which means the first KeepAlive packet does not get sent until 2 hours later. For networks that disconnect idle connections within e.g. 15 minutes, this is too late. To reduce this to e.g. 5 minutes:
echo 300 > /proc/sys/net/ipv4/tcp_keepalive_time
Note: this change will not survive reboot. To make this change permanent the /etc/sysctl.conf needs to be edited.

After restarting Tomcat, the connection will look like this:
# netstat -anpo | grep :80
tcp6    0      0 192.168.1.7:80    123.123.123.123:38123   ESTABLISHED 21345/java      keepalive (82.52/0/0)

References:

Friday, December 9, 2016

Ansible 2.2 with_items Deprecation

Prior to 2.2, this worked
- name: do something
# do something here
with_items: ec2.instances

However, this has been deprecated. Changelog item: "with_ 'bare variable' handling, now loop items must always be templated {{ }} or they will be considered as plain strings"

It's now required to do this:
- name: do something
# do something here
with_items: "{{ ec2.instances }}"


References:

Friday, September 30, 2016

MongoDB "root" Role

To perform functions such as replication administration, the "dbAdminAnyDatabase" role is not enough. The "root" role is required".
db.grantRolesToUser("admin",[{role:"root","db":"admin"}])

Reference: http://stackoverflow.com/questions/23943651/mongodb-admin-user-not-authorized

Ansible Move File/Folder in Idempotent Manner

Solution is to use conditionals:

       - name: stat the dest path
         stat: path=/path/dest
         register: statdest
       - name: move file
         command: mv /path/src /path/dest
         when: statdest.stat.exists == false

Reference: http://stackoverflow.com/a/24165405/548272

Thursday, September 1, 2016

Bash: Easy Way to Tokenize Strings Separated by Space

Using the "set" command, it is possible to replace the position parameters with the one assigned. This can then be used to refer to individual elements of a string separated by space.

$ str="one two three"
$ set -- $str
$ echo $1
one
$ echo $2
two
$ echo $3
three

Reference: http://stackoverflow.com/questions/5382712/bash-how-to-tokenize-a-string-variable

Ansible Relative Path with playbook_dir

The playbook_dir resolves to the directory where the playbook file resides. You can use relative path from there to reference other files.

Ansible Variables

Good resource about variables which compiles all the different ways variables are used in Ansible: https://liquidat.wordpress.com/2016/01/26/howto-introduction-to-ansible-variables/

Ansible Variables from File

In the task e.g.
- hosts: appservers
 vars_files:
       - vars.yml

vars.yml e.g.:
var1Name: var1Value
var2Name: var2Value
var3Name: var3Value

References:

Wednesday, August 31, 2016

AWS IAM Policy Statement to Register and Deregister Instances with/from ELB

E.g.

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "Stmtxxxxxxxxx",
           "Effect": "Allow",
           "Action": [
               "elasticloadbalancing:RegisterInstancesWithLoadBalancer",
               "elasticloadbalancing:DeregisterInstancesFromLoadBalancer"
           ],
           "Resource": [
               "arn:aws:elasticloadbalancing:aws-region:aws-account-id:loadbalancer/*"
           ]
       }
   ]
}

AWS IAM Permissions to Create Tags

Had a hard time getting this to work, and it turns out that only permissions to the "ec2:CreateTags" action is needed, however, the moment the resource is constrained (even to all resources within a specific region), I was unable to tag an EC2 instance. I was only able to tag an EC2 instance when the "Resource" was set to *

{
   "Version": "2012-10-17",
   "Statement": [
       {
           "Sid": "Stmtxxxxxxxx",
           "Effect": "Allow",
           "Action": [
               "ec2:CreateTags"
           ],
           "Resource": [
               "*"
           ]
       }
   ]
}