Introduction to DevOps

Published an older presentation I provided on

https://www.slideshare.net/Ram0603/devops-a-peek-into-high-performing-organizations-108817405

Install pip on AWS EC2

While stock EC2 AMI comes with python installed, pip version is pretty old. Updating pip results in pip going out of path - it gets installed on a different location.

Here is sample script to update pip and still keep pip running:

#!/bin/bash
yum update -y
pip install -U pip
export PATH=$PATH:/usr/local/bin

Please make sure that it is run as root user only, otherwise it will error out with permission error

SVN: Error running context: An error occurred during authentication

Problem Statement: My corporate laptop suddenly started giving "Error: Error running context: An error occurred during authentication" on all Tortoise SVN requests. I switched to subclipse, and I got same. I switched to command line, and I got same!

Root Cause: My corporate password had changed and somehow all SVN clients were using the old password.

Fix:
I see multiple fixes mentioned online, eg http://stackoverflow.com/questions/914895/how-to-change-credentials-for-svn-repository-in-eclipse and http://www.wandisco.com/svnforum/forum/opensource-subversion-forums/general-setup-and-troubleshooting/tortoisesvn-community-support/12445-basic-authentication-not-working and others, but 

DEVOPS: Automatically updating job parameter in Jenkins

This post would ideally follow up after a discussion on configuring Jenkins jobs with Sprint number as a parameter to build jobs. At the start of every sprint, that Sprint number needs to be updated to new sprint. While I'll save the philosophy on that for a later discussion, here's whats here

Problem Statement:
A Jenkins job parameter needs to be updated every 'x' days. This needs to be done for 'n' jobs. A default way is to do it manually, an obviously error-prone way of doing it, where we run the risk of making a wrong update, as well as miss one of the 'n' jobs. In the following config XML of a Jenkins job, the Sprint number needs to be incremented as part of this job

<hudson.model.StringParameterDefinition>
    <name>SPRINT</name>
    <description></description>
    <defaultValue>10</defaultValue>
</hudson.model.StringParameterDefinition>

Resolution:
In true spirit of automation, this task can be automated by creating a Jenkins job that looks at these jobs and updates each of them. Here is a sample shell script (hey, Jenkins is running on Unix, right ;), and can be easily customized for powershell script.

#!/bin/bash
# This job uses Jenkins REST API to fetch job config, invokes python to increment SPRINT
#   and again uses Jenkins REST API to publish updated job config
# It defines an array of jobs that needs to be updated
# It defines a python script that will update job configurations
# It defines a shell function that will compile python function and pass jenkins config, and execute it
# The updated config is POST'ed back to Jenkins


JOB_NAMES=(first-build-job second-build-job third-build-job)
# Step 1: assign python code to increment Sprint value to a shell variable
_increment_sprint_script=$(cat <<'EOF'
import sys, xml.etree.ElementTree as ET
doc = ET.fromstring(sys.stdin.read())
for node in doc.findall('.//hudson.model.StringParameterDefinition'):
    name_el = node.find('./name')
    if name_el is not None and name_el.text == 'SPRINT':
        default_el = node.find('./defaultValue')
        if default_el is None: continue
        default_el.text = str(int(default_el.text) + 1)
print ET.tostring(doc)
EOF
)


# Step 2: define a function that calls the interpreter with that code
increment_sprint() { python -c "$_increment_sprint_script" "$@"; }


# Step 3: Iterate over all jobs and update them
for JOB_NAME in "${JOB_NAMES[@]}"
do
  echo "Updating $JOB_NAME ..."
  updated_config=$(curl "$JENKINS_URL/job/$JOB_NAME/config.xml" | increment_sprint)
  curl -v -X POST --data-binary "$updated_config" -H 'Content-Type: application/xml' -u "$USERID:$TOKEN" "$JENKINS_URL/job/$JOB_NAME/config.xml";
done

The finer details:
1. Notice the quotations in curl commands - pretty important, miss these, and the Jenkins configuration of x-build-job will turn into a single line xml, blotching its script.
2. Its easy to get the configuration wrong by a simpler script that just looks for value of "defaultValue" of current Sprint. While this may usually work, it can fail if there are multiple parameters with same defaultValue as current Sprint





Externalizing system properties or truststore

The need, aka Why: 

Set system properties based on environment. While there are many ways to externalize properties in java world (JNDI), spring profile, or as Stack Overflow search reveals, JBoss System Properties; there aren't many ways to externalize system properties - other than setting them up in .profile or similar file. However, if you need a consistent and more clear way of setting them up, there aren't as many options (or I am not aware of).

A solution, aka What:

A clear, clean and intuitive ( and I'd say smart) solution is to let Spring handle it for you. Yep, it can be done, and pretty easily too!

The following code sets the system property javax.net.ssl.truststore to specify location of truststore, defined in a standard property file that is easily configured per-environment. It is searchable in editor and consistent across operating systems, so is kinda neat.

Of course, if you are setting the truststore, you might need to set the trust store password as well - which would not be plain text property, and hence not such a simple task, but thats for another day ;)

How:

Well, the code is pretty self-explanatory, Spring uses Map / property bean titled "systemProperties" to store the system properties. The above code uses Spring's MethodInvokingFactoryBean to modify this bean to add additional properties to System properties. At least that's what I believe is happening.

Challenges:

The above solution worked for me when the truststore was in classpath, otherwise on Windows 8, it gave me a weird InvalidAlgorithmParameterException. I looked at SO but that doesnt help me, though it looks like a generally correct response. Well, again I'll leave that for another day!

Spring JMS - Tibco EMS Integration

Recently I had to publish 100+ messages on our Tibco EMS to test out behavior of application server that consumes them, with different configurations. Surprisingly, there are no easy to use blogs on using Tibco EMS with Spring JMS to send messages. So, I had to scramble and here is generalized code inspired from that work. Obviously, this is far from production ready.

In this maven example we will read message from pre-defined location in file system and send to a Tibco EMS Queue.

application-context.xml


MessagePublisher.java


pom.xml

Java Reinvented | Salient Features of Java 8 | Part 1

Java 8 Salient Features

The 2013 Java One keynote says, "Java is Back'. Not only is it back, it is new, it is easier to read, and at the same time it is difficult to learn, specially if you are used to coding in Java. Multiple brilliant features have been added in Java 8, a quick introduction of 52 features was presented at Devoxx and is available at Parleys.

This blog is first part of a series on some of the features that I believe would be of significant value to everyday developer. Most of them are related to lambda expressions itself, which is going to significantly alter coding in Java. The following will be covered:

1. Lambda Expressions
2. Default Methods
3. Functional Interface
4. Streams and Bulk Collections
5. Date and Time API
6. Base 64 Encoding Schemes
7. Permgen removed, permanently

Lambda Expressions

Also known as closures, a lambda expression is an anonymous method - has an argument list, a return type and a body. A lambda expression looks like a problem statement and hence code is much easier to understand. eg, to print names of all male employees whose first name contains letter 'n', sorted by their names, the following code can be used.

As you can see, re-written in Lambda, code becomes so much simpler. Additionally, instead of two "for Each" passes, lambda code needs a single pass (with some caveats).

Default Methods
Modifying an existing, widely used, mature language like Java is no easy task. So when Java developers were working on lambda expressions, they faced a wall in extending existing well known packages. The solution came through modifying interface contract to support providing default implementation of a method within interface itself. Adding default methods in an interface is source and binary compatible.

A default method is also known as defender method or virtual extension method. The following image shows clip of Iterable interface in JDK 8. As you can see, the "remove" method is now "default" allowing developers to not implement it in their classes that implement Iterable interface.

An additional method "forEachRemaining" has been added (details suppressed for brevity), allowing existing implementation to not be broken by addition of new method.

Java folks say developers would not need to worry about diamond problem due to this, as this adds behaviour inheritance only, and not data inheritance - that usually causes issues. Additionally, as earlier, compile time checks prevent a class from implementing two interfaces with same method.

Adding behaviour (methods) to definition (interface) may not be a really good idea, but hopefully this will resolve more problems than it creates.

Look out for the next post, showing examples of more features. All references will be part of last post :)