New Project Launched: Spectacular.Gift
It began like this: Amazon.com’s vast product catalog contains many clever and unique items, the sort that you may not know you wanted until you’ve heard of it. Alternately, these items might make an ideal gift when shopping for the person “who already has everything”. So I figured it would be a neat idea to curate a collection of these items and build a gift recommendation site around them. Doing so would allow me to explore some new server-side technologies and help keep my skills fresh.
Technologies used:
- Ubuntu VPS from Linode
- Apache2 HTTP server
- Apache Tomcat 7
- Apache ActiveMQ
- Spring 3.1 with Spring MVC
- PostgreSQL 9.3
- HTML5 + CSS3
- jQuery, flot, TinyMCE
- Java libraries such as Joda Time, Jsoup, Jackson, Logback, and Commons DBCP
- Amazon Product Advertising API
- Reddit REST API
To get some data populated in the database as a starting point, I set up a scheduled task to pull data from several Reddit forums where Amazon links are shared. Reddit conveniently makes this data available via their REST API. All products discovered in this way are set to unapproved status pending manual review.
Next, I set up another scheduled task to populate and refresh metadata about the products via Amazon’s Product Advertising API. Per Amazon’s terms in order to display price information, this data has to be refreshed hourly. For efficiency I request data on batches of ten products at a time, which is the maximum limit.
I created a manual process for reviewing and approving products to be shown. This process includes writing a custom description, adding relevant tags (e.g. “For Kids” or “Gadget Lovers”), and setting an age range and gender specificity (if applicable).
The UI is written in JSP and outputs HTML5. Some features are powered by javascript, such as the price history button which uses flot to render the graph of historical price data.
Spring 3.1 ties it all together. Spring MVC handles the front-end. Spring JDBC is used for interacting with PostgreSQL. I could have used Spring’s event system, but I wanted to get some experience with ActiveMQ. There are a number of message senders and listeners set up for events such as “price changed” or “product suggested”.
I’ll probably think of a snappier name eventually, but for now I registered http://spectacular.gift (new “.gift” TLD). Have a look if you like! It’s basically in beta, and I’m still adding new products and tags.
Log SQL Statements With Parameter Values Filled In (Spring JDBC)
If you’re using parameterized queries with Spring JDBC (and why wouldn’t you?), it’s easy to log them with a simple log.debug() statement. But your logged statements will be full of ?’s instead of the values, which makes them much less useful. Suppose you’d like to know what was substituted for those question marks. It’s not so difficult if you use AspectJ. Here is what is needed:
- aspectjrt.jar and aspectjweaver.jar from here.
- An aspect with a pointcut on the JdbcOperations interface, which
JdbcTemplateimplements. @Beforeadvice that intercepts the execution ofJdbcTemplatemethods and logs de-parameterized SQL statements.- Configuration of Spring applicationContext.xml to get it working.
Let’s start off with our class:
@Aspect
public class SqlLogger {
private static final Logger log = LoggerFactory.getLogger(SqlLogger.class);
Here, I’m using an org.slf4j.Logger, but any logging framework will work.
Next step is to add the method that will capture the SQL and parameters as they are executed. Again, I’m using Spring JDBC, so all calls are made to an object that implements JdbcOperations. We can set up our class to spy on all relevant calls by defining our method like so:
Configure Lucene IndexWriter and IndexSearcher in Spring applicationContext.xml
Problem: you want to define Lucene IndexWriter and IndexSearcher as beans inside your Spring application to be injected/autowired into other beans.
Solution: follow the following steps.
- define the Lucene version as a constant
- define a Lucene analyzer (StandardAnalyzer) as a bean
- define a Lucene directory as a bean, using a factory-method for instantiation
- define an IndexWriter, wiring in the Lucene directory and an IndexWriterConfig set to use your previously-defined analyzer
- define an IndexSearcher, wiring in the Lucene directory
- define also a query parser (StandardQueryParser), wiring in the analyzer bean
You can then wire/autowire these beans into your application beans, for example:
Read more
Web scraping in Java with Jsoup, Part 2 (How-to)
Web scraping refers to programmatically downloading a page and traversing its DOM to extract the data you are interested in. I wrote a parser class in Java to perform the web scraping for my blog analyzer project. In Part 1 of this how-to I explained how I set up the calling mechanism for executing the parser against blog URLs. Here, I explain the parser class itself.
But before getting into the code, it is important to take note of the HTML structure of the document that will be parsed. The pages of The Dish are quite heavy–full of menus and javascript and other stuff, but the area of interest is the set of blog posts themselves. This example shows the HTML structure of each blog post on The Dish:
Read more
Web scraping in Java with Jsoup, Part 1
In order to obtain the data to feed into my blog analyzer, content must be parsed from the pages of the blog itself. This is called “web scraping”. Jsoup will be used to parse the pages, and because this is a Spring project, Spring scheduling will be used to invoke the parser.
The following classes were created:
- BlogRequest – invokes the parser on a given blog URL, passes parsed content to service layer
- BlogRequestQueue – queues up and executes blog requests
- BlogParser – interface with parseURL method
- DishBlogParser – implements BlogParser, used to parse the blog The Dish
Each of these (aside from the interface) is configured as a Spring-managed bean. The code for BlogRequest:
Read more
Profile SQL statements in Java / Spring
Wouldn’t it be nice if there were a way to time your application’s SQL statements unobtrusively? This information could give you insight into the performance of your queries and updates and help you identify slow, poorly-performing SQL. Of course, there is a way to add such SQL profiling to your Spring application, by using AspectJ.
I use Spring JDBC and wanted to identify slow SQL queries in my application so that I could tune them in order to improve overall performance. Capturing the execution times whenever SQL is executed can be done by creating a pointcut on the methods of JdbcTemplate. Here is what we need:
- aspectjrt.jar and aspectjweaver.jar from here.
- An aspect with a pointcut on the JdbcOperations interface, which JdbcTemplate implements.
- @Around advice that times the execution of JdbcTemplate methods and stores this data for later retrieval.
- Configuration of Spring applicationContext.xml to get it working.
The pointcut looks like this, with the String argument being the SQL statement:
Read more
Add custom annotation to Spring MVC controller
The question of how to add custom annotations to my Spring MVC controllers puzzled me for some time, because the documentation in this area is lacking. Even the bible of Spring 3.0, Spring Recipes by Gary Mak, et. al.
But then I found this great blog post detailing exactly what I was interested in. In a nutshell, you need to implement WebArgumentResolver and set your class as a customArgumentResolver of the AnnotationMethodHandlerAdapter bean. What I was interested in was adding a @RequestAttribute annotation that would work like @RequestParam, but would obviously pull the value from a request attribute rather than a request parameter.
Read more
JSON / Ajax in Spring MVC
How do you configure your Spring MVC web application to serve JSON for Ajax? It is not difficult. You probably have a view resolver in your dispatcher-servlet.xml that looks like this:
<bean id="viewResolver" class="org.springframework.web.servlet.view.InternalResourceViewResolver" p:viewClass="org.springframework.web.servlet.view.JstlView" p:prefix="/WEB-INF/jsp/" p:suffix=".jsp"></bean> |
In order to serve JSON content, you need to use the ContentNegotiatingViewResolver. The configuration looks like this:
Read more
Handy RowMapper base class for Spring JDBC
RowMappers are needed all over the place in your Spring JDBC DAO classes. One challenge that I kept running into was that when I wanted to reuse a particular RowMapper class for numerous queries, there was an ever-present threat of an underlying SQLException if certain columns were not present in the ResultSet. Obviously, a reusable RowMapper will set every field on the object it maps for, however not every ResultSet will include every field. Calling rs.getString("column_name"); will result in an exception being thrown if column_name is not present in the particular ResultSet.
So to solve this problem, I wrote this base RowMapper class:
Read more
Define global lists, sets and maps in Spring configuration
Sometimes you may need to define stand-alone collections–lists, maps and sets–in your Spring XML configuration files, so that your beans can reference them as properties. This is easy to do using the util schema. For example, let’s say you want to define a set of credit scores in your applicationContext.xml file:
<util:set id="creditScores"> <value>600</value> <value>710</value> <value>760</value> </util:set> |
Now you can reference the set in your beans, like so:
<bean id="creditBean" class="com.this.is.my.CreditBean"> <property name="myScores"> <ref local="creditScores"/> </property> </bean> |
