Ansible 2.2 with_items Deprecation

Prior to 2.2, this worked

- name: do something # do something here with_items: ec2.instances

However, this has been deprecated. Changelog item: "with_ 'bare variable' handling, now loop items must always be templated {{ }} or they will be considered as plain strings"

It's now required to do this:

- name: do something # do something here with_items: "{{ ec2.instances }}"

References:

• https://github.com/ansible/ansible/blob/devel/CHANGELOG.md#22-the-battle-of-evermore---active-development
• https://github.com/ansible/ansible/blob/devel/CHANGELOG.md#removed-deprecated

MongoDB: Setup Replica Set

Simple guide: https://blog.ajduke.in/2013/05/31/setup-mongodb-replica-set-in-4-steps/

Additional steps with keyfile auth:

• https://docs.mongodb.com/v2.6/tutorial/deploy-replica-set-with-auth/
• https://docs.mongodb.com/v3.2/tutorial/enforce-keyfile-access-control-in-existing-replica-set/

MongoDB "root" Role

To perform functions such as replication administration, the "dbAdminAnyDatabase" role is not enough. The "root" role is required".

db.grantRolesToUser("admin",[{role:"root","db":"admin"}])

Reference: http://stackoverflow.com/questions/23943651/mongodb-admin-user-not-authorized

Ansible Move File/Folder in Idempotent Manner

Solution is to use conditionals:

- name: stat the dest path          stat: path=/path/dest          register: statdest        - name: move file          command: mv /path/src /path/dest          when: statdest.stat.exists == false

Reference: http://stackoverflow.com/a/24165405/548272

Bash: Easy Way to Tokenize Strings Separated by Space

Using the "set" command, it is possible to replace the position parameters with the one assigned. This can then be used to refer to individual elements of a string separated by space.

$ str="one two three" $ set -- $str $ echo $1 one $ echo $2 two $ echo $3 three

Reference: http://stackoverflow.com/questions/5382712/bash-how-to-tokenize-a-string-variable

Ansible Relative Path with playbook_dir

The playbook_dir resolves to the directory where the playbook file resides. You can use relative path from there to reference other files.

Ansible Variables

Good resource about variables which compiles all the different ways variables are used in Ansible: https://liquidat.wordpress.com/2016/01/26/howto-introduction-to-ansible-variables/

Ansible Variables from File

In the task e.g.

- hosts: appservers  vars_files:        - vars.yml

vars.yml e.g.:

var1Name: var1Value var2Name: var2Value var3Name: var3Value

References:

• https://www.safaribooksonline.com/library/view/ansible-up-and/9781491915318/ch04.html
• http://docs.ansible.com/ansible/playbooks_variables.html
• http://stackoverflow.com/questions/27746077/how-to-include-vars-file-in-a-vars-file-with-ansible

AWS IAM Policy Statement to Register and Deregister Instances with/from ELB

E.g.

{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "Stmtxxxxxxxxx",            "Effect": "Allow",            "Action": [                "elasticloadbalancing:RegisterInstancesWithLoadBalancer",                "elasticloadbalancing:DeregisterInstancesFromLoadBalancer"            ],            "Resource": [                "arn:aws:elasticloadbalancing:aws-region:aws-account-id:loadbalancer/*"            ]        }    ] }

AWS IAM Permissions to Create Tags

Had a hard time getting this to work, and it turns out that only permissions to the "ec2:CreateTags" action is needed, however, the moment the resource is constrained (even to all resources within a specific region), I was unable to tag an EC2 instance. I was only able to tag an EC2 instance when the "Resource" was set to *

{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "Stmtxxxxxxxx",            "Effect": "Allow",            "Action": [                "ec2:CreateTags"            ],            "Resource": [                "*"            ]        }    ] }

Ansible Best Practices: Directory Layout

Here: http://docs.ansible.com/ansible/playbooks_best_practices.html#directory-layout

AWS CloudWatch Logs: Create Log Stream via Command Line

aws logs create-log-stream --log-group-name groupName --log-stream-name streamName

Ansible: Running a Command on the Local Machine

To run an command on the shell of Ansible machine (i.e. instead of the target machine), use the local_action module:

local_action: command xxxxxxxxx

Ansible: EC2 Instance ID Variable

{{ ansible_ec2_instance_id }}

MongoDB: Query for Documents Based On Array Size

db.collectionName.count({"arrayName":{$exists:true},$where:'this.arrayName.length>3'})

Reference: http://stackoverflow.com/questions/7811163/query-for-documents-where-array-size-is-greater-than-1

Ant Java Compilation Classpath

If there are many jar files in the classpath, instead of specifying all of them, include all jar files in a folder this way: http://stackoverflow.com/questions/6103516/ant-adding-multiple-jars-in-classpath-dynamically

Amazon Linux Install Ansible

The YUM method won't work, due to differing python versions.

Use pip instead.

sudo pip install ansible

Passing Environment Variables into Java from Command Line

java -DvarName1=varValue1 -DvarName2=varValue2

Pentaho Data Integration PostgreSQL Bulk Loader Gotcha

The password you key in the connection won't work. It only works when authentication is disabled for the server Pentaho is running on.

Reference: http://wiki.pentaho.com/display/EAI/PostgreSQL+Bulk+Loader#PostgreSQLBulkLoader-Setupauthentication

Cheat Sheet on 16 Analytics Techniques

Here it is: http://www.forbes.com/sites/bernardmarr/2016/02/04/the-18-best-analytics-tools-every-business-manager-should-know/#ebb1f7d2c4ad

MongoDB "update" Only Updates One Entry by Default

The "update" command only updates the first matching entry by default. To update all entries, there is a third parameter i.e. "options" where you need to set "multi: true" e.g.

db.coll.update({},{$set:{"attr":”value”]}},{multi:true})

Reference: https://docs.mongodb.org/v2.4/reference/method/db.collection.update/

AWS IAM: EC2 Full Access to Region

Example:

{    "Version": "2012-10-17",    "Statement": [        {            "Sid": "xxxxxxxxxxx",            "Effect": "Allow",            "Action": [                "ec2:*"            ],            "Resource": "*",            "Condition": {            "StringEquals": {                "ec2:Region": "us-west-2"            }        }      }    ] }

Reference: http://stackoverflow.com/questions/18112784/iam-allowing-a-user-to-access-everything-for-ec2-on-a-region

AWS S3: Setup IP Based Access Control

Use the "Condition" attribute:

"Condition" : {    "IpAddress" : {        "aws:SourceIp": "xxx.xxx.xxx..xxx/32"    }, }

More examples can be found in following references:

• http://docs.aws.amazon.com/AmazonS3/latest/dev/amazon-s3-policy-keys.html
• https://pete.wtf/2012/05/01/how-to-setup-aws-s3-access-from-specific-ips/

Techcrunch 2015 Year End Analysis of the Analytics Market

Great read!

http://techcrunch.com/2016/01/09/the-year-in-analytics/

Disable Android File Transfer on MacOS

I seldom transfer file so this is an unnecessary annoyance.

Steps here: https://cooltrainer.org/taming-android-file-transfer-on-mac-os-x/

MongoDB Security

By default, security and authentication is disabled. Create the administrator account before switching on authentication. Enter the console of the admin DB:

mongo admin

Add the administrator user:

db.createUser({user: "adminUserName",pwd:"adminUserPassword",    roles: [ { role: "userAdminAnyDatabase", db: "admin" }, { role: "readWriteAnyDatabase", db: "admin" }, { role: "dbAdminAnyDatabase", db: "admin" }, ] })

Enable authentication by editing the "/etc/mongod.conf" file
Uncomment this line:

auth=true

Restart MongoDB. Enter the console of the admin DB and you'll find that you'll need to authenticate before performing any operations:

db.auth({user: "adminUserName",pwd:"adminUserPassword"})

Now you should be able to add a user for the app database containing your data, and make this user the owner of the app database:

db.createUser({user: "dbUserName",pwd:"dbUserPassword",    roles: [ { role: "dbOwner", db: "dbName" }] })

Note: this needs to be done inside the app database

To enable remote login, comment out the "bind_ip" in "/etc/mongod.conf".

Note: Steps tested on MongoDB 2.6.x

AWS Ant Upload to S3

Use AWS Ant Tasks, an open source library released by AWS in the AWS Labs.

To use it, define the taskdef:

<taskdef resource="taskdefs.xml" classpath="~/.ant/lib/aws-java-sdk-ant-tasks-1.2.3.jar" />

Example of upload task:

<target name="deploy-s3" depends="war">    <upload-to-s3 bucketName="bucket-name" keyPrefix="folder-name/" continueOnFail="false" awsAccessKeyId="${aws.access.key}" awsSecretKey="${aws.secret.key}">        <fileset>            <!-- define files here -->        </fileset>    </upload-to-s3> </target>

Note: As at current time of writing it's broken with Java 8. You'll have to run Ant with Java 7.

References:

• https://github.com/awslabs/aws-ant-tasks
• https://java.awsblog.com/post/Tx1VW8X2W3775XC/AWS-Ant-Tasks

AWS Command Line Download S3 File

If you want to use the save credentials in ~/.aws/credentials, use the "aws s3 cp" command:

aws s3 cp s3://<bucket-name>/<folder-name>/<file-name> <local-file-name>

Reference: http://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Setting Up AWS Command Line

Do this in order not to have to repeat credentials and region for every command.

aws configure

There is one "gotcha" though. Commands like ec2-start-instances won't use the saved information and you'll have to provide the credentials still. In order to use the saved information, you'll have to call the API via the "aws" command e.g.

aws ec2 start-instances --instance-ids i-xxxxx i-xxxx2 i-xxxxx3

Reference: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

AWS Start EC2 Instance from Command Line

First, setup permissions. The simpler way to do it is to use IAM, create a group, give it permissions to start instances and then add a user to the group.

The permission can be added for each instance using the Policy Generator:

1. Effect: Allow
2. AWS Service: Amazon EC2
3. Actions: StartInstances
4. ARN: e.g. arn:aws:ec2:ap-southeast-1:<accountId>:instance/<instanceId>

To include multiple instances, either add multiple statements or edit the policy statement manually later to add more instances.

You can start the instance via AWS API using the "ec2-start-instances" command.

References:

• http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-StartInstances.html

AWS ARN Reference

At this page: http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html

AWS Find Account ID

Apparently it's necessary to go to the support page to find it!

http://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html

Linux Can Ping But Cannot Traceroute

This is because Ping uses ICMP but Traceroute sometimes uses UDP by default. To use Traceroute with ICMP use the "-I" option.

Reference: http://superuser.com/questions/278952/why-can-i-ping-an-ip-address-but-not-traceroute-it

Pentaho Data Integration: Unwinding JSON Array Elements From MongoDB

This can be done in the MongoDB Input step. However, it only supports one level of nesting.

To do that, use "arrayName[*]" as part of the path.

MongoDB Query Return Multiple Entries for Array Elements

A JSON document stored in MongoDB is tree-based. If it contains multiple sub-elements, they will have to be unwound first before processing.

One way to do that is with the "$unwind" operator in the aggregation framework.

Examples here: https://docs.mongodb.org/v2.4/reference/operator/aggregation/unwind/