Apache HttpClient connect

Introduction

 As sometimes needed you want one service to communicate with the other without interference of the frontend. One of the ways to do that is to setup a connection through the apache httpClient. In our project we allready did that but with an old version of te HttpClient. Therefore we needed to upgrade to the latest version. At this point in time it is 4.5.1. The only downside of this client is, that it is still not upgraded to HTTP 2.0. read the specs here. But we did it anyway in the hope that the coming version would use the HTTP 2.0 specifications. 

The solutution

This solution is not giving you the whole shabang  but it gives you a good idea howto setup a basic configuration that works and will help you to setup through a proxy or not.

//The maven part:

<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpclient</artifactId>
    <version>4.5.1</version>
</dependency>
<dependency>
    <groupId>org.apache.httpcomponents</groupId>
    <artifactId>httpcore</artifactId>
    <version>4.4.4</version>
</dependency>

//If you in need of a proxy set it up like this with ofcourse the right adress and 

//port.

HttpHost proxy = new HttpHost("proxy", 8080, "http");
// Configure the request with timeouts and in this case a proxy. 

//If you don't need a proxy you can leave that particular part out 

RequestConfig config = RequestConfig.custom()
        .setConnectionRequestTimeout(30000)
        .setSocketTimeout(30000)
        .setProxy(proxy)
        .build();

//Instead of using the HttpMethod in the old version we now 

//should use the HttpGet or the HttpPost HttpGet httpget = 

new HttpGet("http://yourservice/getEverything");
httpget.setConfig(config);

//This is one of the ways to setup a HttpClient.

//there are many more ways as you will find in these examples. 

HttpClient client = HttpClientBuilder.create()
        .build();

// retrieving the response as HttpResponse and printout

//the http statusHttpResponse response = client.execute(httpget);
System.out.println(response.getStatusLine());

Conclusion

It is not so hard to set this up once you have an idea of what you need to do. I been searching the internet for a some examples. The sad story of this search was that there a lot of old examples out there. I only hope that this library is soon updated to http 2.0.

retrieve the end date from a certificate

 Introduction

This week we had a really nice challenge. As most companies we use certificates. They help us to keep the connections secure. The downside of certificates is that they need to be renewed now and then. It is so easy to notice an expired certificate by undesirable behaviour of the connection. If this happens  your already to late. To be ahead of this, we decided to do a check up on this particular certificate. 

The solution

To start of all you need an object of the type Certificate. In this mather I start with an certificate, which needed to be read from disk:   

//Take the file from disk as an inputStream using the classloader for this.

InputStream inputStream = this.getClass().getClassLoader().getResourceAsStream("testCertificate/testCertificate.pem");

//getting the CertificateFactory of the type x.509 

//(X509Certificate is the only child of Certificate)

CertificateFactory certFactory = CertificateFactory.getInstance("X.509");
Certificate certificate = certFactory.generateCertificate(inputStream);

// setting up the keystore with an default type. You can also use:

//getInstance(String type,Provider provider)
//getInstance(String type, String provider) // Type could be pkcs12 or jks these are archive file formats for storing many //cryptography objects as a single file. // types could be SUN or bc (bouncy castle).

KeyStore keyStore = KeyStore.getInstance(KeyStore.getDefaultType());

//adding the certificate to the keystore with an alias

keyStore.setCertificateEntry("alias", certificate);

// creating the date object outside the scope of the try we can return it later.

Date notAfter = null;

try {
  // initializing the keystore with null to prevent exceptions to be thrown later on.

  keyStore.load(null);

//checking if the keystore has an alias and retrieve the first one.

 //(we only have one). If you know the alias you can loose this piece of code.

    if(keyStore.aliases().hasMoreElements()) {
        Enumeration<String> aliasses = keyStore.aliases();
        String alias = aliasses.nextElement(); 

// cast it to the X509Certificate That has the getNotAfter method. Certificate

//doesn't have that.

X509Certificate certificate = (X509Certificate) keyStore.getCertificate(alias);

 // assigning the date to the end date of the certificate.

notAfter = certificate.getNotAfter();
    }
} catch (KeyStoreException ke) {
    log.error("could not read keystore or certificate", ke);
} catch (CertificateException ce) {
    log.error("no certifications in keystore", ce);
} catch (NoSuchAlgorithmException nse) {
    log.error("wrong algorythm for certificate.", nse);
} catch (IOException ioe) {
    log.error("could not read certificate.", ioe);
}

// returning the end date.

return notAfter;

Conclusion

To understand what is happening in this particular piece of code acquires some time and dedication. The fun part starts when you understand it. I still enjoy the moments where I crack a case like this and understand why it is needed to  be done like this. I hope you enjoy it also.

Have fun!

testing log statements

Introduction

Testing can sometimes make your head spin. For example when a method is a void and you are not allowed to use powermock. 

The thing with a method like that, is that it is not returning an Object on which you can test on. 

If you are lucky it might have a parameter that changes during the execution of the method. 

Sometimes it has a log statement based on a Exception or on an event captured in an if statement, that happened during execution.  

That gives also opportunities. Let me show how.

The solution

 

For working with log files the best thing to do is approach them with log appenders. 

//We start with a mock appender: 

@Mockprivate Appender mockAppender;
//Creating a rootlogger to hold all the appenders. 
ch.qos.logback.classic.Logger root = 
(ch.qos.logback.classic.Logger) LoggerFactory.getLogger 
(ch.qos.logback.classic.Logger.ROOT_LOGGER_NAME);
 
 // appending the ArgumentMatcher to the mockAppender. 
Mockito.verify(mockAppender).doAppend(Mockito.argThat 
(new ArgumentMatcher<Object>() { 
//implementing the matches method to see if the
// log text appears in the log file 
@Overridepublic boolean matches(final Object argument) {
return ((LoggingEvent) argument).getFormattedMessage()
//The text we want to search. 
.contains("test logging text that should be found");}}));
 

The conclusion

Void methods could be still a call for an asperine when it comes to testing. There are some helpful methods I believe that this is one of them. I used it this time to test a method that checked if an certificate was expired. This way I could pump up the test coverage of my code.

Have fun!

 

A bit of Reflection

Introduction

In all the years I have been programming one of the dullest things I found to do was writing and maintaining converters. I had a couple of discussion about the subject. The point that always came up as a defence to not use bean mappers like Dozer was the fact that they used Reflection under the hood. I agree up to some level that the use of Reflection in production code should not be used in production code. The interesting point is that most of the third party libraries are using Reflection under the hood. Every library that uses Annotations for example are using it. Think about Spring, all the JPA libraries, log4j, Junit etc...
So why not for something like bean mapping. So lets see if we can build one our selfs.

The solution

A bean mapper is meant to  copy values from one bean to the other. So what you need to accomplish this is to know the field names of the "other" bean.

 The solution is actually not that complicated What I did was create a couple of Annotations and a handler.

If the field names are different you can pass the name of the fields in the "other" bean by using an annotation.

The Annotations look like this:

//The rentention policy tells when this Annotation should be used.

@Retention(RetentionPolicy.RUNTIME)

//The target tells where the annation should be placed. In this case field only.
@Target(ElementType.FIELD)
public @interface Copy {

    // the string that contains the value given to the Annotation.
    public String copyTo();

}

If the field names are the same you can do it without an annotation. 

I show that later.

If you dont want the fields to be copied you can use simply another annotation that prevents the field to be part of the copy process:

@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.FIELD)
public @interface IgnoreCopy {
}

The annotation in use look something like this:

// copying with a different field name

@Copy(copyTo = "testString2")
private String testString;
// copying wit the same fieldname
private String testStringNoAnnotation;

should not be copied.
@IgnoreCopyprivate String notToCopy;

To copy the values first of all you need the fields of the bean to copy from:

//Use the getDeclaredFields instead of the getFields. Otherwise you can't find 

//the field you are searching for. 

Field[] fields = copyFrom.getClass().getDeclaredFields();

//looping through the fields to find the annotation if needed.

for(Field field : fields) {

    //The annotation if the field names are different.
    Copy copy = field.getAnnotation(Copy.class);

    //The annotation to see if the field shouldn't be copied. 
    IgnoreCopy ignoreCopy = field.getAnnotation(IgnoreCopy.class);
    if(ignoreCopy == null) {
        if ( copy != null) {

            //the actual method to copy with different field name 
            addValueToField(copyFrom,copyTo, field, copy.copyTo());
        } else {
            //the actual method to copy with the same  field name 
            addValueToField(copyFrom, copyTo, field, field.getName());
        }
    }
}

private void addValueToField(final Object copyFrom, 

final Object copyTo, final Field field, final String name) {
    try {

        // Searching for the field in the "other" bean
        Field copyToField = copyTo.getClass().getDeclaredField(name);

        // The fields are private so we make them accessible
        copyToField.setAccessible(true);
        field.setAccessible(true);

        // The actual copying of the values
        copyToField.set(copyTo, field.get(copyFrom));

        //stop the accessibility of the fields. 
        copyToField.setAccessible(false);
        field.setAccessible(false);
    } catch (NoSuchFieldException e) {
        log.error("could not find field " + name);
        e.printStackTrace();
    } catch (IllegalAccessException e) {
        log.error("could not access field " + name);
        e.printStackTrace();
    }
}

// I added two more methods that are helpfull with the cherry picking. 

/** * checks if ignorefield is set in the target.class 

* @param copyTo the target class. 

* @param fieldName the fieldname to be set. 

* @return a boolean if the field is right to set. 

*/

 private boolean ignoreCopyToField(final Object copyTo, final String fieldName) {
    Field field = null;

    // A list of fieldnames to prevent the NoSuchFieldException

         List<String> fieldNames = checkFieldExists(copyTo);
        IgnoreCopy ignoreCopy = null;
        try {

            // check if the field exists in the target class. 

           if (fieldNames.contains(fieldName)) {

                // retrieve the field with the name 

                field = copyTo.getClass().getDeclaredField(fieldName);

                // retrieve the annotation from the field
                ignoreCopy = field.getAnnotation(IgnoreCopy.class);
            }
        } catch (NoSuchFieldException e) {
            e.printStackTrace();
        }        return  ignoreCopy != null;
}

/** * creates a list that has all the fieldnames in it. 

* @param copyTo the target class 

* @return a list that has all the fieldnames in it 

*/ 

private List<String> checkFieldExists(Object copyTo) {
    List<String> fieldNames = new ArrayList<String>();
    Field[] fields = copyTo.getClass().getDeclaredFields();
    for (Field field1: fields) {
        fieldNames.add(field1.getName());
    }
    return fieldNames;
}
 

Conclusion

The choice to be made is more of a philosophic one. Is it worth to add some more reflection to the project at hand. I think the decision is to use it where it is useful. If it can save you writing and maintaining miles of boring code my advice would be to do it. In many cases it allready happens. So think about and work this out for your self. The code to build this is short and simple.

The original source code can be downloaded  here

The jar file can ben downloaded  here

Have fun!

Sorting data

Introduction:

In the issue I had at hand we needed to sort the data to create a matrix. So actually we needed to morph a list to a matrix form. 

The problems:

At first hand I was thinking about using the states of the data to figure out where in the matrix it should find its place. As soon I made a drawing about my idea, I stopped working at it. I needed to create a complex structure with a lot of if statements. I understood that, if I choose this way I might run into a lot of bugs. I needed to think harder. 

The solution:

I turned the whole idea around and looked at it from the other side. Instead of creating the matrix directly from the list I could something like a pre-sort. The next question obvisiouly would be: How the hell do I do that? The answer was actually quite simple looking back at the issue. I took a LinkedhashMap and a Treemap. I made sure that the query returned everything sorted by date. 

In the LinkedHashmap I used as a key the date and the Treemap. The LinkedHashmap keeps things in order of incoming. So it kept the order of the query I created by date. The second ordering would be by currency  where the payment was in done.  This I used as a key in the TreeMap together with the rest of the object. The Treemap sorts on aliphatic or numeric order using the black and red sorting algorithm. 

Now All I had to do was write a double loop for both of the maps and put the data I needed in place. 

Conclusion:

Sometimes it is better to make that drawing and have a better look at what you need to-do. I believe that part is the greatest joy in my job. Create something that gives me the proud feeling.

Have fun!

saving passwords the save way

Introduction:

Every time I search on security matters, I always find the things to prevent attacks from the outside. These issues are important there is no doubt about that. But the one issue that kept running around my mind was one of the jobs where they got into your database. That is the place where an unwanted visitor should stay away. One of the last resort solutions to prevent this unwanted visitor to steal your customer data is to hash the passwords and encrypt the data in that database. The downside of this technique is that most of the encrypting and hashing methods all ready have white papers. So the only thing is: that buys you a bit more time before they got to your data.

Part of the solution:

I did not solve the whole case of steeling data yet. I did found away to store passwords strongly protected in my database. The solution is quite simple: don't store the real password in the database but an abstraction of that password. Every character has an ASCII number changing the characters in the password into numbers gives you the possibility to calculate with them like this:


double key = 0;
char[] passwordChars= password.toCharArray();
for (char keyPart : passwordChars) {
      key += Character.getNumericValue(keyPart);
}

Now you have a nummeric value of a password. How far you will calculate this is upto you.
The next question is ofcourse what to do with a number. The simple thing is that you can allready store this number into the database and you have an abstraction of the password. But the fun is to push it a bit further. You can create an hashmap that has a double as a key and a String as a value.
With the calculated number you get the string value out of the database. and store that as a password.

Something like this:
private Map<Double, String> falsePasswords = new HashMap<Double, String>();
and fill it with nice long strings with a lot of weird characters:
falsePasswords.put(1.0, "*(_)((UIiuyuUITYFTYR%R&$%&%tituyutyr867987yuyuiyo8&)*0");
I would make this a lon list of false passwords and create a calculation on the password that gives a good range of false passwords.

To make it look like a real password you can hash it and make the unwanted visitor think that he can decrypt all your passwords.

Conclusion:

The solution to most of the problems are not that hard. In this case a simple calculation and a hashmap brought the solution.

Have fun!

Spring-boot rest/jpa full blown

Introduction

For a while I was looking for a suiting solution where I could use spring rest services combined with the Spring/JPA repositories. What I actual was looking for was a query driven rest service where I did not had to convert one bean to another. This awfull boilerplate code and is a maintainance nightmare.

After a lot of searching I ran into Spring-boot where the got a couple of steps further than I was looking for. After studying it for a while I got used to their ideas and must say I like them.

While I was searching I ran into a lot of examples but none of them was showing me how to deal with spring-boot in a professional and serious mather. So i picked up a lot from the reference guide and put the pieces of the puzzle together.

The configuration
I build the whole story in Maven for the gradle fans have a look at the reference guide. There is fully explained howto do it in gradle.

The basiscs of spring-boot is the simple fact that they provide you with a super pom for maven where you can cherry pick the parts you really like to use. Here are the cherries I picked from that tree:

The project pom 
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <parent>
        <artifactId>abstractservice</artifactId>
        <groupId>com.purat.services</groupId>
        <version>1.0-SNAPSHOT</version>
    </parent>
    <modelVersion>4.0.0</modelVersion>
    <artifactId>userservice</artifactId>
    <packaging>jar</packaging>

    <properties>
        <tomcat.version>8.0.3</tcat.version>
        <!-- use UTF-8 for everything -->
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <start-class>com.purat.Application</start-class>
        <spring-boot-version>1.1.10.RELEASE</spring-boot-version>
    </properties>
// The super pom provided by spring-boot
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>1.1.10.RELEASE</version>
    </parent>

    <dependencies>
// these are the depedency to deliver all the necessary jpa libraries to approach the database. 
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-jpa</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jdbc</artifactId>
        </dependency
//This is the library to deliver all the necassary rest libraries. To create for example your own endpoints
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-data-rest</artifactId>
        </dependency>
//This is the library that gives us the web application part. 
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
//This is the library that gives us the production-ready parts. For more details look at boot-starter-actuator
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
//For the database I used postgress.
        <dependency>
            <groupId>postgresql</groupId>
            <artifactId>postgresql</artifactId>
            <version>9.0-801.jdbc4</version>
        </dependency>
// For handling the getters and setters I used lombock.
        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
                <version>1.12.6</version>
                <scope>compile</scope>
        </dependency>
    </dependencies>   <build>
        <pluginManagement>
        <plugins>
//This plugin is responsible for the kind of packaging you desire. You can choose  a war or a jar
This plugin will build it with the desired content.  
To deploy a war file there are still a couple of steps to take. You can find the description for that here:
traditional deployment
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>3.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                </configuration>
            </plugin>
        </plugins>
        </pluginManagement>
    </build>

    <repositories>
        <repository>
            <id>spring-releases</id>
            <name>Spring Releases</name>
            <url>https://repo.spring.io/libs-release</url>
        </repository>
        <repository>
            <id>org.jboss.repository.releases</id>
            <name>JBoss Maven Release Repository</name>
            <url>https://repository.jboss.org/nexus/content/repositories/releases</url>
        </repository>
    </repositories>

    <pluginRepositories>
        <pluginRepository>
            <id>spring-releases</id>
            <name>Spring Releases</name>
            <url>https://repo.spring.io/libs-release</url>
        </pluginRepository>
    </pluginRepositories>
</project>

The Application class and application.properties

Despite of what you see on the internet I like to keep the Application class to a bare minimum. We need this baby only to start the project up after we build a war or a jar. 

import org.springframework.boot.SpringApplication;

import org.springframework.boot.autoconfigure.EnableAutoConfiguration;

import org.springframework.context.ConfigurableApplicationContext;

import org.springframework.context.annotation.ComponentScan;

import org.springframework.context.annotation.Configuration;

/**

 * Created by compurat on 1/11/15.

 */

//Indicates that a class declares one or more @Bean methods and may be processed by the Spring container to generate bean definitions and service requests for those beans at runtime.

@Configuration

// This annotation configures  and uses the standard beans that are on the classpath of the diverse libraries.

@EnableAutoConfiguration

//Configures component scanning directives for use with @Configuration classes.

@ComponentScan

public class Application {

    public static void main(String[] args) {

//This is the only line you need to start up you application.

        ConfigurableApplicationContext context = SpringApplication.run(Application.class);

    }

}

The application.properties file under the resources folder:

The way this works, looks to me like the Wicket framework. 

//The configuration of the database

spring.datasource.url=jdbc:postgresql://localhost:5432/people

spring.datasource.username=Pieter

spring.datasource.password=Pieter01

spring.datasource.driver-class-name=org.postgresql.Driver

spring.jpa.database-platform=org.hibernate.dialect.PostgreSQLDialect

hibernate.hbm2ddl.auto=create-drop

spring.jpa.show-sql=true

// Setting the end point properties give it an id,give the possibility to shut the endpoints down and sensitive is security wise. Does an endpoint need username and password to be approached.

endpoints.beans.id=springbeans

endpoints.beans.sensitive=false

endpoints.shutdown.enabled=true

//Tomcat serverlog enabling and patterns.

server.tomcat.access-log-enabled=true

server.tomcat.access-log-pattern=%a asdasd

The log file propeties.

logging.file= logging/userservice.log

logging.level.org.springframework.web: DEBUG

logging.level.org.hibernate: ERROR

Basicly we now have ready to run spring-boot webservice with JPA. It does not work yet we still need to fill in the missing pieces here. 

The application

Lets start with the rest endpoint. That is where it all begins. This is the profile endpoint and does not take any parameters in this case. But it give you an idea how an endpoint in spring-boot works:

import com.purat.data.People;

import com.purat.data.repository.PeopleRepository;

import org.springframework.beans.factory.annotation.Autowired;

import org.springframework.web.bind.annotation.RequestMapping;

import org.springframework.web.bind.annotation.RestController;

/**

 * Created by compurat on 1/11/15.

 */

//Tells spring that this is obvious a rest controller

@RestController

//Tells spring what the base url is.

@RequestMapping("peopleservice")

public class ProfileEndpoint {

    @Autowired

//spring/jpa repository to approsch the database

    private PeopleRepository peopleRepository;

//Tells spring what the endpoint url is.

    @RequestMapping("/profile")

    public Person profile() {

   The jpa entity

    Person person 

//The query parameter names are in the repository method name. in this case EmailadressAndPassword

= peopleRepository.findByEmailadressAndPassword("pieter.roskam@gmail.com","Roskam01");

If you Return the entity in the rest controller spring will underwater serialize the object to an json file.

        return person;

    }

} 

As second part I like to handle the jpa entity:

import lombok.Getter;

import lombok.Setter;

import javax.persistence.Entity;

import javax.persistence.Id;

/**

 * Created by compurat on 1/11/15.

 */

//Lombock getters and setters keep you code nice and clean.

@Getter

@Setter

//What table to approach

@Entity(name="people")

public class Person {

// the unique id and all the fields as named in the table.

    @Id

    private long id;

    private String firstname;

    private String lastname;

    private String address;

    private String housenumber;

    private String postalcode;

    private String city;

    private String telephone;

    private String verification;

    private String emailadress;

    private String password;

}

As third and last part I would like to show the jpa repository:

package com.purat.data.repository;

import com.purat.data.User;

import org.springframework.data.repository.CrudRepository;

 

The spring/jpa repository gives some benefits. For simple queries you can use the parameter names in the method, but for more complicated queries you can also use the @Query annotation to write a jql query .

public interface PeopleRepository extends CrudRepository<Person,Long> {

    User findByEmailadressAndPassword(String emailadress, String password);

}

Conclusion

Spring-boot took a complete new road to developing and deploying rest services. The most interesting part is probably that you can have a rest service in a jar file. That was a mind boggle to me for a while. The secret lies in the fact that it is a self executing jar. While starting up your Application class it will also startup tomcat if you configured it right (dependencies and properties file).

The second part that might be of interest, is the fact that it looks like that the separation of tiers is gone. But if you look closely it is not:

1. The separation of the database is still intact because it is hidden behind the spring/jpa repositories. They are not reachable from the outside. 

2. The entity bean seams to be exposed at the frontend but again that is not completely true. The bean will be serialized to a json file by Spring. That is the only visible thing on the outside. The only one that should reach this service is the frontend. Which is also separate.

I love the part that you can package a rest service into a jar. You put it somewhere on your server and start it with java -jar servicewhatever.jar. That will fire up everything you build for this rest service.

have fun!