shopping24 tech blog

s is for shopping

February 25, 2015 / by Kim Hogeling / Web developer / @kimhogeling

shopping24 Hacking Days Winter 2014

2 days in the last week of the year again gave room to explore some of the ideas and projects the shopping24 developers had. We do this on a regular schedule, last time in August 2014.

image

For hacking days, we encourage the tech team members to challenge new technologies or technologies new to them. This time, among some internal others, the followin projects got chosen.

Traffic replay with Gor

Most of our performance problems are only reproducable within the production environment where real users work with the system. A simple ui test, a JUnit test or an integration test is not sufficient to detect these problems beforehand. Load test tools like JMeter are a valid way to detect performance problems. The ultimate way to test an application is nevertheless to expose it to real user traffic.

gor-image

A traffic replay tool as gor can record real user traffic and replay it on a staging system. This seems to be a good solution to be as near to a production environment as possible. We have build an ansible task to provision a server with gor, starting traffic recording and downloading the recorded traffic afterwords. The next step is to automate traffic replay on a staging system.

Developer Site

Our developer site – the site you are staring at right now – did not always look like this. It used to look way less modern. Right before the hacking days to be precise! As most of you probably already noticed, we embraced the so called “Material Design”. Because this new design emphasizes all element groups, the site looks cleaned up a lot now. Attention is drawn to more important content.

We already did use Twitter Bootstrap for the site, so we simply started the redesign by adding a material design theme. The GitHub user FezVrasta created one, which did the trick: https://github.com/FezVrasta/bootstrap-material-design

The rest of the endresult came out of playing around. We tried different colours, both back- and foreground and moved around some of the elements. We also made the right navigation accessible for mobile users. Now we can really call the site responsive! And lastly something that we find extremely important; We applied a few performance best practises for a (little) performance boost.

image image

When’s that product online? Measuring end-to-end product data import

Writing tests is part of our normal development process. But so far, we have not had a comprehensive, automated test for the end-to-end data flow from feed to search index, which spans multiple applications. During the hacking days, we have written a simple toolset to perform this automated end-to-end testing. We wanted the tools to cover two kinds of test cases:

  • Measure the end-to-end time of our import data flow.
  • Better support for semi-automated tests of import configuration settings such as filters, data mappings, etc.

We implemented two tools that operate at each end of our import data flow. The first tool is a test feed generator which, as the name might suggest, generates a test feed. The second tool is a Solr observer which searches for the products on our Solr servers. In addition to those tools, we have of course also added a query to our existing monitoring infrastruture to monitor the results of the end-to-end timing measurements.

To measure the end-to-end import time, the feed generator simply adds a new product to the test feed at regular intervals and records this action and a timestamp in a database. The Solr observer queries the database for products that were added to the test feed but have not been observed in the search results yet. As soon as a product is found in the search results, the database is updated. By comparing creation and oberservation dates we can then measure the overall import time.

Better handling of Solr “descriptions”

On every e-commerce system there is a problem which you have to face with someday: Bad Data at your products. When we’re looking through our product-descriptions, we always find SEO-texts or very unspecific and not descriptive sentences. These sentences needs to be removed from the description to get better search results. So we build a Solr-Tokenizer that filters bad sentences from texts that does not contain any relevant information.

As an example we want to reduce the following input:

"Dieses Cocktailkleid begeistert mit seiner edlen Farbnuance, funkelnden Pailletten 
und dem zarten Spiel mit Transparenz an den Ärmeln. Wasserfallausschnitt 3/4 Arm
nahtfeiner Reißverschluss in der hinteren Mitte Gesamtlänge: 154 cm Länge ab 
Taille: 112 cm"

The result is very impressive and the tokenizer performs very well:

image

Mapping-Badges

We have an internal tool to normalize all the partner shop categories to our own category tree - everyone in the company could use it - we call it Mapping-Mammut. The Mapping-Mammut already has a daily, weekly and monthly leaderboard - but how can we motivate our colleagues to map even more? - even more gamification! :) We implemented achievements and added a nice overview with bronze, silver, gold and platin medals. Now we have a few achievements which will be checked after every mapping (eg “you have mapped x things”) and a few which will be checked by several crons (eg “you are a daily/monthly… hero”, “allTimeHero”, “you reached the topTen“ …).

Github Issues Geeboard

We recently moved from our Redmin bugtracker to Github issues](github.com) for a single project. For issue handling this is great, but the need for a swimlane-like overview of issues, features and status increased. So the idea was to build a kanban board for Github Issues.

A nice board with different lanes. Each lane stands for a specific status of a collection of tickets. I decieded to use Angularjs to build this fancy app. The Github API is well documented as well as AngularJs is. I started from scratch and at the end of the two days, I was able to get all of our github issues from the API as well as the meta information of our labels. Additionally t he drag and drop feature to move the tickets from lane to lane was ready. AngularJs as JS-Framework was the right decision. There are a lot built in functions to have a very fast start. Looking forward to go one with this project in the future.

Want to hack with us?

We are hiring!