AWS S3: Setup IP Based Access Control

Use the "Condition" attribute:

"Condition" : {    "IpAddress" : {        "aws:SourceIp": "xxx.xxx.xxx..xxx/32"    }, }

More examples can be found in following references:

• http://docs.aws.amazon.com/AmazonS3/latest/dev/amazon-s3-policy-keys.html
• https://pete.wtf/2012/05/01/how-to-setup-aws-s3-access-from-specific-ips/

Techcrunch 2015 Year End Analysis of the Analytics Market

Great read!

http://techcrunch.com/2016/01/09/the-year-in-analytics/

Disable Android File Transfer on MacOS

I seldom transfer file so this is an unnecessary annoyance.

Steps here: https://cooltrainer.org/taming-android-file-transfer-on-mac-os-x/

MongoDB Security

By default, security and authentication is disabled. Create the administrator account before switching on authentication. Enter the console of the admin DB:

mongo admin

Add the administrator user:

db.createUser({user: "adminUserName",pwd:"adminUserPassword",    roles: [ { role: "userAdminAnyDatabase", db: "admin" }, { role: "readWriteAnyDatabase", db: "admin" }, { role: "dbAdminAnyDatabase", db: "admin" }, ] })

Enable authentication by editing the "/etc/mongod.conf" file
Uncomment this line:

auth=true

Restart MongoDB. Enter the console of the admin DB and you'll find that you'll need to authenticate before performing any operations:

db.auth({user: "adminUserName",pwd:"adminUserPassword"})

Now you should be able to add a user for the app database containing your data, and make this user the owner of the app database:

db.createUser({user: "dbUserName",pwd:"dbUserPassword",    roles: [ { role: "dbOwner", db: "dbName" }] })

Note: this needs to be done inside the app database

To enable remote login, comment out the "bind_ip" in "/etc/mongod.conf".

Note: Steps tested on MongoDB 2.6.x

AWS Ant Upload to S3

Use AWS Ant Tasks, an open source library released by AWS in the AWS Labs.

To use it, define the taskdef:

<taskdef resource="taskdefs.xml" classpath="~/.ant/lib/aws-java-sdk-ant-tasks-1.2.3.jar" />

Example of upload task:

<target name="deploy-s3" depends="war">    <upload-to-s3 bucketName="bucket-name" keyPrefix="folder-name/" continueOnFail="false" awsAccessKeyId="${aws.access.key}" awsSecretKey="${aws.secret.key}">        <fileset>            <!-- define files here -->        </fileset>    </upload-to-s3> </target>

Note: As at current time of writing it's broken with Java 8. You'll have to run Ant with Java 7.

References:

• https://github.com/awslabs/aws-ant-tasks
• https://java.awsblog.com/post/Tx1VW8X2W3775XC/AWS-Ant-Tasks

AWS Command Line Download S3 File

If you want to use the save credentials in ~/.aws/credentials, use the "aws s3 cp" command:

aws s3 cp s3://<bucket-name>/<folder-name>/<file-name> <local-file-name>

Reference: http://docs.aws.amazon.com/cli/latest/reference/s3/cp.html

Setting Up AWS Command Line

Do this in order not to have to repeat credentials and region for every command.

aws configure

There is one "gotcha" though. Commands like ec2-start-instances won't use the saved information and you'll have to provide the credentials still. In order to use the saved information, you'll have to call the API via the "aws" command e.g.

aws ec2 start-instances --instance-ids i-xxxxx i-xxxx2 i-xxxxx3

Reference: http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html

AWS Start EC2 Instance from Command Line

First, setup permissions. The simpler way to do it is to use IAM, create a group, give it permissions to start instances and then add a user to the group.

The permission can be added for each instance using the Policy Generator:

1. Effect: Allow
2. AWS Service: Amazon EC2
3. Actions: StartInstances
4. ARN: e.g. arn:aws:ec2:ap-southeast-1:<accountId>:instance/<instanceId>

To include multiple instances, either add multiple statements or edit the policy statement manually later to add more instances.

You can start the instance via AWS API using the "ec2-start-instances" command.

References:

• http://docs.aws.amazon.com/AWSEC2/latest/CommandLineReference/ApiReference-cmd-StartInstances.html

AWS ARN Reference

At this page: http://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html

AWS Find Account ID

Apparently it's necessary to go to the support page to find it!

http://docs.aws.amazon.com/IAM/latest/UserGuide/console_account-alias.html

Linux Can Ping But Cannot Traceroute

This is because Ping uses ICMP but Traceroute sometimes uses UDP by default. To use Traceroute with ICMP use the "-I" option.

Reference: http://superuser.com/questions/278952/why-can-i-ping-an-ip-address-but-not-traceroute-it

Pentaho Data Integration: Unwinding JSON Array Elements From MongoDB

This can be done in the MongoDB Input step. However, it only supports one level of nesting.

To do that, use "arrayName[*]" as part of the path.

MongoDB Query Return Multiple Entries for Array Elements

A JSON document stored in MongoDB is tree-based. If it contains multiple sub-elements, they will have to be unwound first before processing.

One way to do that is with the "$unwind" operator in the aggregation framework.

Examples here: https://docs.mongodb.org/v2.4/reference/operator/aggregation/unwind/