Archive

Archive for the ‘code’ Category

XOP Converter

December 18, 2010 2 comments

My most recent task at work involves XOP, or XML-binary Optimized Packaging. Basically, if you have an XML document that needs to have binary data in it, like images, you have two ways of doing it. You can base64 encode the data and put it directly in an XML element, or you can use XOP to keep the data in binary form and attach the file to the message using MIME. XOP is usually used to represent SOAP messages with MTOM (Message Transmission Optimization Mechanism).

My app uses JiBX to convert my Java objects into XML documents, and base64 encodes them. But I got a new requirement to also support XOP encoding. So to minimize impact on the app, I decided to write a converter that would take in a XML document with base64 encoded data and spit out a properly encoded XOP document. Then it would also be useful to convert existing documents and creating new ones. The module that helped me do this was Apache Axiom, a component of the Apache Axis2 web services engine.

The main problem with Axiom is its poor documentation. Almost all of the docs I could find about Axiom and Axis2 were from a server-side perspective, and I am writing a client-side app. So I thought I’d share the bit of code I wrote to do this conversion.

public class XOPConverter {

    @SuppressWarnings("rawtypes")
    public static void convertToXOP(String xml, OutputStream os) 
        throws XMLStreamException, JaxenException, JiBXException, IOException {

        // set up all the AXIOM objects
        XMLInputFactory factory = XMLInputFactory.newInstance();
        XMLStreamReader parser = factory.createXMLStreamReader(new StringReader(xml));
        StAXOMBuilder builder = new StAXOMBuilder(parser);
        OMDocument document = builder.getDocument();
        OMElement documentElement =  builder.getDocumentElement();
        OMFactory fac = OMAbstractFactory.getOMFactory();
        
        // find all the BinaryBase64Object nodes
        AXIOMXPath xpathExpression = new AXIOMXPath("//niem-nc:BinaryBase64Object");
        xpathExpression.addNamespace("niem-nc", "http://niem.gov/niem/niem-core/2.0");
        List nodeList = xpathExpression.selectNodes(documentElement);
        
        for (Iterator i = nodeList.iterator(); i.hasNext(); ) {
            Object obj = i.next();
            if (!(obj instanceof OMElement)) {
                continue;
            }
            OMElement element = (OMElement)obj;
            // get bytes data out of that node
            String base64data = element.getText();
            // decode base 64 encoding
            byte[] data = Base64Serializer.deserializeBase64(base64data);
            // create ByteArrayDataSource
            ByteArrayDataSource ds = new ByteArrayDataSource(data);
            // set the MIME type
            ds.setType("image/" + XOPConverter.getMIMEType(data));
            // create OMText object
            OMText textElement = fac.createOMText(new DataHandler(ds), true);
            // replace BinaryBase64Object node text with OMText object
            element.setText("");
            element.addChild(textElement);
        }
        
        // do the XOP conversion
        OMOutputFormat format = new OMOutputFormat();
        format.setDoOptimize(true);
        format.setMimeBoundary("MIME");
        format.setContentType("Multipart/Related");
        MTOMXMLStreamWriter mtomWriter = new MTOMXMLStreamWriter(os, format);
        mtomWriter.setDoOptimize(true);
        document.serializeAndConsume(mtomWriter);
        document.close(false);
    }
    
    /**
     * @param args
     * @throws IOException 
     * @throws XMLStreamException 
     * @throws JaxenException 
     * @throws JiBXException 
     */
    public static void main(String[] args) 
        throws IOException, XMLStreamException, JaxenException, JiBXException {
        
        if (args.length == 0) {
            System.out.println("Usage: <filename>");
            return;
        }
        String inputFileName = args[0];
        String outputFileName = inputFileName.substring(0, 
                inputFileName.lastIndexOf(".")) + ".xop";
        boolean overwrite = false;
        if (args.length > 1) {
            String overwriteStr = args[1];
            overwrite = Boolean.parseBoolean(overwriteStr);
        }
        // check if output file exists
        if (new java.io.File(outputFileName).exists() && !overwrite) {
            System.out.println("ERROR: Output file " + outputFileName + 
                    " exists, please move it.");
            return;
        }
        
        java.io.BufferedReader reader = 
            new java.io.BufferedReader(new java.io.FileReader(args[0]));
        String line = null;
        StringBuilder stringBuilder = new StringBuilder();
        String ls = System.getProperty("line.separator");
        while ((line = reader.readLine()) != null) {
            stringBuilder.append(line);
            stringBuilder.append(ls);
        }
        String xml = stringBuilder.toString();
        
        FileOutputStream os = new FileOutputStream(outputFileName);
        convertToXOP(xml, os);
        System.out.println("done");
    }
}

Now I’ll step through the code and explain it a bit.

The convertToXOP method takes the XML text as a String, and an OutputStream where the XOP is written. In my original document, all the base64 encoded objects I need to extract and attach are in elements labeled . So I cook up an XPath expression to find all those elements for processing. The code gets the data out of the element and un-base64 encodes it. I didn’t write that code, so you’ll have to do that yourself. Then we put the byte array in a ByteArrayDataSource and tag it with the MIME type we want to use. In a later post I’ll show how to figure out what encoding an image uses. Then use the OMFactory to create a new OMText element containing the byte array and set it to be optimized. Add it to the original element and reset its text, clearing out the base64 encoded data.

The key to doing the conversion is Axiom’s MTOMXMLStreamWriter class. You send the MTOMXMLStreamWriter the OutputStream where you want the XOP document to go, and an OMOutputFormat object. Setting the doOptimize flag on the OMOutputFormat object enables the optimization step that moves the binary objects to MIME attachments. Sending the MTOMXMLStreamWriter to the OMDocument’s serializeAndConsume method writes the XOP output.

I hope this code and explanation has been helpful to you.

Categories: code Tags: , ,

Scala test drive

November 22, 2010 Leave a comment

Last week I got together with some coworkers to learn Scala and write some code. Since it was our first gathering, we kept it very simple and tried to really understand how the Scala code works and how it is different from Java.

Before the session, my day job gave me a very simple programming task, to read first and last names out of a directory of xml files and write them to a CSV file along with the name of the xml file it came from. There are many better and easier ways to do this, but I wrote it in Java, intending to rewrite it in Scala with my group.

Here’s the Java code:

import java.io.File;
import java.io.FileOutputStream;
import java.io.FileFilter;
import java.io.IOException;
import java.io.OutputStreamWriter;
import javax.xml.parsers.DocumentBuilder;
import javax.xml.parsers.DocumentBuilderFactory;
import javax.xml.parsers.ParserConfigurationException;
import javax.xml.xpath.XPath;
import javax.xml.xpath.XPathConstants;
import javax.xml.xpath.XPathExpressionException;
import javax.xml.xpath.XPathFactory;
import org.w3c.dom.Document;
import org.w3c.dom.NodeList;
import org.xml.sax.SAXException;

public class OTHExtractor {
    private static OutputStreamWriter writer = null;
    
    public static void main(String[] args) throws ParserConfigurationException,
            SAXException, IOException, XPathExpressionException {
        writer = new OutputStreamWriter(new FileOutputStream(new File("names.java.txt")));
        try {
            File[] files = new File(".").listFiles(new FileFilter() {
                public boolean accept(File file) { 
                    return file.getName().endsWith(".xml");
                } 
            });
            for (File file : files) {
                process(file);
            }
        } finally {
            writer.flush();
            writer.close();
        }
        System.out.println("done");
    }

    private static void process(File xmlFile)
            throws ParserConfigurationException, SAXException, IOException,
            XPathExpressionException {
        DocumentBuilderFactory dbf = DocumentBuilderFactory.newInstance();
        DocumentBuilder db = dbf.newDocumentBuilder();
        Document doc = db.parse(xmlFile);

        XPathFactory xpFactory = XPathFactory.newInstance();
        XPath xpath = xpFactory.newXPath();

        NodeList nodeList = (NodeList)xpath.evaluate("//PersonGivenName/text()", 
                doc, XPathConstants.NODESET);
        String givenName = "";
        if (nodeList != null && nodeList.getLength() > 0) {
            givenName = nodeList.item(0).getNodeValue();
        }

        nodeList = (NodeList)xpath.evaluate("//PersonSurName/text()", 
                doc, XPathConstants.NODESET);
        String surName = "";
        if (nodeList != null && nodeList.getLength() > 0) {
            surName = nodeList.item(0).getNodeValue();
        }
        writer.write(givenName + ", " + surName + ", " + xmlFile.getName() + "\n");
    }
}

And here’s the Scala code we wrote to replace it:

import java.io.PrintWriter
import scala.xml.XML
import java.io.File

object OTHExtractorScala {
  def main(args:Array[String]) = {
    val root = new File(".")
    
    val files = root.listFiles().filter(file => file.getName.endsWith(".xml"))
    val writer = new PrintWriter(new File("names.txt"))
    try {
      files foreach (file => writer.write(process(file)))
    } finally {
      writer.flush
      writer.close
    }
    println("done")
  }
  
  def process(file:File) = {
    val names = parseXML(file)
    names._1 + ", " + names._2 + ", " + names._3 + "\n"
  }
  
  def parseXML(file:File) = {
    val xml = XML.loadFile(file)
    ((xml\\"PersonGivenName").text, (xml\\"PersonSurName").text, file.getName)
  }
}

That’s 64 lines of verbose code whittled down to 29 very readable lines. Not bad!

Let’s take a closer look at some of this fancy Scala code.

    val files = root listFiles() filter(file => file.getName.endsWith(".xml"))

The first interesting thing about this line of code is that the call to filter the result of root’s listFiles call. We skipped the dots! You’re allowed to skip the dot and the parentheses when you call a function with zero or one arguments. Here we had to include the parentheses after listFiles so Scala would know that we’re calling a function called filter and not sending a value called filter to the listFiles function.

Next up is file => file.getName.endsWith(".xml")). What is this? This is an anonymous function. The bit before the => is the argument this function takes, in this case file. The part after the => is the body of the function. The Scala Array object’s filter method takes a function that returns Boolean. So this is a much slicker way of getting just the xml files than Java’s FileFilter interface.

The other thing I want to point out is the tuple. The parseXML method returns a tuple where the first element is the given name, the second element is the surname, and the third element is the file name. In Java we’d have to package them into a collection of some sort, or create an inner class to hold the values. Tuples make this much easier. The process method grabs the elements from the tuple names and turns them into a concatenated String separated by commas.

Oh, one other thing. Scala’s XML module ROCKS. Look how much easier it is to run XPath queries on documents. One line of code creates the XML document, one line of code finds the elements in the document. We needed a lot more verbose Java to do that.

Categories: code Tags:

eclipse.ini

November 3, 2010 Leave a comment

Eclipse is great, but it takes a while to get started on my work laptop. I decided to search around to see what I could do to speed it up.

The first tip I found was to make sure you were running Eclipse under the most recent JDK. By default Eclipse uses the first JVM it finds in your path. The server components on this project require Java 5, not 6, so many developers have to configure their systems to allow Maven to use Java 5. They can still install Java 6 and use it with Eclipse with the following argument:

-vm "C:\Program Files\Java\jdk1.6.0_22\bin\javaw.exe"

You can either put that in the shortcut you use to start Eclipse, or you can put it in the Eclipse.ini file.

The second tip is to disable unnecessary plug-ins on startup. In Eclipse 3.6 (Helios) this is configurable in Preferences -> General -> Startup and Shutdown. I don’t think a lot of people use Mylyn, so that’s a feature to disable if you don’t.

The third tip is to customize the eclipse.ini file. I found this great post on StackOverflow that explains the best options for entries in eclipse.ini for any recent version of Eclipse you are using.

For an in-depth look at all the Eclipse runtime options, check out the Eclipse documentation.

Categories: code Tags:

Local Variables

November 3, 2010 Leave a comment

A few years ago a colleague criticized some code I wrote for local variable inefficiency. Here’s an example of what he did not like:

for (int i = 0; i < 10; i++)
{
	String s = String.valueOf(i);
}

He said that Java allocated 10 Strings when that code ran. A better way was this:

String s;
for (int i = 0; i < 10; i++)
{
	s = String.valueOf(i);
}

So that’s the way I’ve been writing Java code for years now, thinking I am being super efficient.

Well, joke’s on me. Today I found this blog post and learned that his assumption is completely incorrect. The first way above is better because it makes the code more readable and prevents possible variable scoping problems.

Categories: code Tags:

Scala

October 29, 2010 Leave a comment

What is Scala?

In their book The Pragmatic Programmer, Andrew Hunt and David Thomas suggest “Learn a new language every year.” I looked around at the hot new languages out there today and decided that Scala was the most promising one to learn.

Scala, short for Scalable Language, is both a functional language and an object-oriented language. Like Groovy and JRuby, Scala runs on the Java Virtual Machine, which means it’s easy to integrate Scala code with existing Java code. It is a statically typed langauge, like Java, which means its type checking is performed during compile-time, as opposed to dynamically typed languages like Groovy and Ruby whose type checking is performed at runtime.

Scala code can be compiled and run as class files, like Java, or you can run it as a script directly from the .scala file. There’s even an interactive command line mode for quickly trying out some code.

Scala has a very concise syntax. It tries to reduce the signal-to-noise ratio of Java code, making the code more readable. It makes a lot of the dots and parentheses of Java code optional, which can be disconcerting for old school Java heads. Clearly the mark of an experienced Scala developer will be the conciseness of their code.

Scala supports a lot of nifty language features like tuples, closures and currying, which makes it a good language to learn what the heck those are and how they might be useful to you in your coding. In general, Scala feels like Java 2.0 in the same way that Java felt like C++ 2.0.

Quick History of Scala

Scala was originally released in late 2003 by its creator, Martin Odersky. The most recent version of Scala is 2.8.0, released in August 2010.

A free web framework called Lift supports Scala code in an environment very similar to Ruby on Rails.

A number of high profile websites have recently switched from Ruby on Rails to Scala & Lift, including Twitter and Foursquare.

James Strachan, the original creator of Groovy, famously said in a blog post in 2009 that if someone had shown him the book Programming In Scala, by Scala’s creator Martin Odersky, he probably wouldn’t have created Groovy.

Programming Scala Book Review

One of Scala’s big draws is its Actor model for concurrent programming. The book I read, Programming Scala by Venkat Subramaniam, showed its focus on this area in its subtitle, “Tackle Multicore Complexity on the Java Virtual Machine”. These days, the new microchips that power our computers feature more and more cores on a single chip. To take advantage of that multicore power, software must be written to take advantage of concurrent processing.

Anyone who’s worked with Java threads knows how painful they can be to get right. Scala’s emphasis on functional programming, and the Actor model, make writing concurrent code a lot easier than in Java. Functional programming avoids state and mutable data, making it ideal for concurrency. The Actor model helps you dispatch functions to perform work and gather the results when each task completes. Anyone needing to write highly scalable code should check out Scala for these valuable benefits.

The rest of Subramaniam’s book provides an excellent overview of Scala. Progamming Scala is part of The Pragmatic Programmer series of books, which strive to be highly readable and concise. At only 208 pages, Programming Scala gets through a lot of material but doesn’t go very deep into any of it. I found myself struggling to grasp the complexity of Scala’s syntax and wished that Subramaniam had gone into more detail in that area.

I’d recommend Programming Scala as a good introduction to the language, but one would probably need to supplement it with a few web based tutorials. I am looking forward to checking out Martin Odersky’s Programming In Scala book soon, which I hope to teach me a lot more about how to write good Scala code.

Categories: code Tags:

Maven & Eclipse

October 29, 2010 Leave a comment

Our project here uses Maven 2 to manage the dependencies and builds. I really like developing code in Eclipse, and I wanted to set up my Eclipse workspace easily from the existing Maven configuration.

The Maven Eclipse plugin makes that easy. Just run the following command:

mvn eclipse:eclipse

and the plugin generates the Eclipse project files, ready to be imported into your Eclipse workspace.

Well, that’s great and all, but when I tried it I saw this error:

[INFO] ------------------------------------------------------------------------
[ERROR] BUILD ERROR
[INFO] ------------------------------------------------------------------------
[INFO] Request to merge when 'filtering' is not identical. Original=source src/test: output=target/test-classes, include=[**/*.java], exclude=[], test=true, filtering=false, merging with=resource src/test: output=target/test-classes, include=[*.properties|*.xml], exclude=[**/*.java], test=true, filtering=true
[INFO] ------------------------------------------------------------------------

A Google search led me to this answer:
http://stackoverflow.com/questions/1397903/setting-project-for-eclipse-using-maven

Basically the problem is that something changed in the latest version of the Maven Eclipse plugin that prevents this from working with our Maven configuration.

The solution is to use the 2.6 version of the plugin instead.

mvn org.apache.maven.plugins:maven-eclipse-plugin:2.6:eclipse

Categories: code Tags: ,

keytool

October 28, 2010 Leave a comment

My most recent project, the Multimodal Operational Test Harness (OTH), is a thick Java Swing client used to gather biometrics and package them up into a properly formatted XML document. The Maven 2 build generates an uber jar including all the dependencies that can be run from the command line, but we also have a requirement to run as a WebStart application. It’s not generally required to digitally sign the software, but we have the requirement locally.

So here’s how I went about signing the OTH.

1) Generate the key pair and self-signed certificate.
I could have used the command line for this step, but instead I decided to use the Maven Keytool plugin.
Here’s a clip of the pom.xml file I created:

<pluginManagement>
	<plugins>
		<plugin>
			<groupId>org.codehaus.mojo</groupId>
			<artifactId>keytool-maven-plugin</artifactId>
			<executions>
				<execution>
					<phase>generate-resources</phase>
					<id>clean</id>
					<goals>
						<goal>clean</goal>
					</goals>
				</execution>
				<execution>
					<phase>generate-resources</phase>
					<id>genkey</id>
					<goals>
						<goal>genkey</goal>
					</goals>
				</execution>
			</executions>
			<configuration>
				<keystore>keystore.jks</keystore>
				<dname>cn=commonName, ou=organizationUnit, o=organizationName, c=countryCode</dname>
				<keypass>password</keypass>
				<storepass>password</storepass>
				<alias>OTH</alias>
				<keyalg>RSA</keyalg>
				<keysize>2048</keysize>
				<validity>3650</validity>
			</configuration>
		</plugin>
	</plugins>
</pluginManagement>

I know it’s not the best idea to put the passwords in the pom.xml file, but the security of this keystore isn’t very important and I wanted to allow other developers to access the keystore without needing to discover the password.

Take special note of the option. If you don’t specify this value, the default is only 90 days. 10 years is much better.

2) Instead of just relying on a self-signed certificate, I need to get my certificate signed by a Certificate Authority. To do that, I generate a certificate signing request (CSR) file.

keytool -certreq -alias oth -file oth.csr -keystore keystore.jks -v

3) I sent that through our security team to our Certificate Authority. They first sent me their root certificates. These have to be imported into our keystore before we can authtenticate the certificate reply.

keytool -importcert -file "rootcert.crt" -alias rootcert -trustcacerts -keystore keystore.jks -v

4) Then I receive my certificate reply from the certificate authority. Now I can import that into my keystore and my certificate is authenticated.

keytool -importcert -file "cert.txt" -trustcacerts -keystore keystore.jks -v

5) You can verify the contents of your keystore by running the following command:

keytool -list -keystore keystore.jks -v

6) Here’s the bit of Maven WebStart plugin configuration that uses the keystore to sign the war.

<plugin>
	<groupId>org.codehaus.mojo.webstart</groupId>
	<artifactId>webstart-maven-plugin</artifactId>
	<version>1.0-beta-1</version>
	<executions>
		<execution>
			<phase>compile</phase>
			<goals>
				<goal>jnlp-download-servlet</goal>
			</goals>
		</execution>
	</executions>
	<configuration>
		<outputDirectory>src/webapp</outputDirectory>
		<excludeTransitive>true</excludeTransitive>
		<jnlpFiles>
			<jnlpFile>
				<templateFilename>jnlp_launch_template.vm</templateFilename>
				<outputFilename>oth.jnlp</outputFilename>
				<jarResources>
					<jarResource>
						<groupId>videology</groupId>
						<artifactId>videology-driver</artifactId>
						<version>1.0.0</version>
						<classifier>win32</classifier>
					</jarResource>
					...
				</jarResources>
			</jnlpFile>
		</jnlpFiles>
		<sign>
			<keystore>${project.basedir}/keystore.jks</keystore>
			<keypass>password</keypass>
			<storepass>password</storepass>
			<alias>OTH</alias>
			<verify>true</verify> <!-- verify that the signing operation succeeded -->
		</sign>
		<!-- BUILDING PROCESS -->
		<pack200>false</pack200>
		<gzip>false</gzip>
		<verbose>true</verbose>
		<outputJarVersions>true</outputJarVersions>
	</configuration>
</plugin>

Finally, here’s the URL of the documentation of the Java keytool tool.
http://download.oracle.com/javase/6/docs/technotes/tools/windows

Categories: code Tags: ,