Ducks keep on looking good from NYC

I’ve pushed a few posts this month beyond midnight, but tonight I almost forgot about my daily responsibility. Tonight’s Duck game was a nail-biter right up to the last second and ended with the Ducks coming back from behind on the road against Arizona.

The Ducks started out looking great. I was feeling pretty confident after getting up by 14 quickly. Then not much happened in the way of good plays for quite a while. To pass disappointing drive after disappointing stand, Rachel made delicious cookies and I ate them.

I am still a little too jittery from all the sugar and the great comeback to sleep, but need to tuck in soon. We have a busy day tomorrow. After a morning walk to the park I have a few duties to get ready for Seth to come for his week-long visit. I also hope to get my ad manager deployed and start running some real traffic through it – but there is a lot to do before that can happen.

My Simple Ad Manager

Swap-bot is in need of a new platform to manage our ads. The previous application we used to manage our inventory recently went through some changes and stopped meeting our needs. As a result, I started looking around for a new system to use. After a some investigation, I decided to build my own and get exactly what we needed instead of adapting to a system that isn’t quite right.

The goal of an ad management platform is to maximize revenue for the publisher, which is done by showing whichever ad will pay the most money at that exact moment to that exact user. That, unfortunately, is much easier said than done. We work with multiple ad networks and have no idea how much they will pay us for each impression until a couple hours or days after it is served. So what do we do?

The basic strategy goes something like this:

  1. Send the first three impressions by a user that day to Network A
  2. Send the next five impressions to Network B
  3. Send the next four impressions to Network C
  4. Send all remaining impressions to Network Z

There currently isn’t a great way to do this. To maximize your revenue takes tedious analysis and lots of trial and error. You never know how many ads a user is going to see, so you want to show the real money makers first.

To see how much each network pays takes time and testing. When evaluating a new network to figure out where in the order they belong you should start by sending it three unique impressions per user per day. After about a week of this you will have a good idea of where the network stands in relation to your other networks. You should run any reports that the new network allows to get a good idea of where you are making most of your money. You should be able to learn which metrics are important to this network and then send it more of what it wants and stop sending it what it doesn’t want.

After doing this for a few years with Swap-bot ads, I have found that geography and obviously frequency are the most important differentiating factors for Swap-bot users. This means that Swap-bot users are similar on any other metrics that are important to advertisers.

Knowing this, I have decided to build a super simple ad manager. To start, my management application is only going to take daily user frequency into account. Even though I know that geography is a factor, I don’t really know where to send the foreign traffic, so adding any sort of geo-decisioning at this point wouldn’t pay off. At some point, I will have enough data to re-evaluate and add any other data decisioning in that would help us make more money.

I plan to talk more about the technical specifics in a later post, but the management app is going to be built using:

  • The Tornado framework for admin ad management and ad tag serving. Tornado is fast to develop in and faster to serve. Even with two datastore look ups per tag request, I’ve got my app up to about 2,000 requests per second on a low-end development machine. Unfortunately, that is more throughput than Swap-bot needs.
  • Nginx for handling incoming requests. Nginx is built on top of libevent and is really fast. Also, since Tornado is single-threaded I will need to run one instance of my app per core and use Nginx to reverse proxy to my four instances.
  • mongoDB for storing possible ads to serve and user cookies. Using mongoDB really makes this an easy project. It has a feature called upsert that makes tracking ad views simple and correct.

In the coming days and weeks I plan to talk more about everything mentioned here. As I build out my new system I will share the decisions I make.

My Two Favorites

Rachel and crusher doing their exercises together.

These two just make we so happy.

New (old) Wilco Records

I’ve been a fan of Wilco since high school. Rachel actually introduced me to them and I have loved every release of theirs since. They recently re-released A.M., Being There and Summerteeth on 180 gram vinyl and my copies just arrived. I have their complete collection an my ipod and listen to it all frequently, but there was still something very exciting about opening the records and setting them each on the turntable for a listen.

I like these early Wilco tracks so much. Part of it is that I actually like the style of play from the early days more – they play with a little more twang and a lot more banjo – but I also have a lot of great memories that involve this music.

I have a hard time deciding which album to listen to, but I am trying to keep them in on even rotation. I am pretty sure they will remain a constant around here until the new Bob Dylan Christmas album arrives.

Instapaper

I did not have my Kindle for my commute this morning so I needed something else to read while on my morning commute (I’ve read online that the easiest way to spot a tourist on the subway is to see who brought something to read and who did not).

I signed up for Instapaper a while ago, but never really used it much. I recently read that Instapaper improved their Kindle support, which renewed my interest in the service, and I had seen some good reviews of the iPhone app.

This morning, before leaving for work, I downloaded the app and it easily loaded up all the articles I had wanted to “read later”. I will probably give the Kindle integration a try, but am very pleased with how the iPhone version worked out for the commute.

If you don’t have a Kindle or iPhone I still suggest you give the service a try. It is super simple to start marking articles to read later and it is even easier to get back to them to read at a later date.

Pizza and the city

Tonight Rachel and I met up with Shane and Alli, who are visiting from Kansas City, for a walk across the Brooklyn Bridge and some of the most famous pizza in all of NYC.

Rachel and I walked across the bridge when we were here visiting before our move, but this was the first time we walked across the bridge at night.

We ate pizza at a place called Grimaldis and the pizza was amazing. The first bite brought a smile to my face. We will definitely be taking anyone that visits us here for at least one meal.

After dinner we walked around DUMBO a bit then parted ways.

On the way home, Rachel and I decided we had to have a little desert so we went to the chocolate room and split fondue.

We are now home, unwinding, and getting ready for tommorrow.



My Wonderful Wife

Rachel does such a great job of making weekends special around here. On Friday when I got home from work, she had delicious, homemade whole wheat chocolate chip cookies in the oven and was in the middle of preparing a fantastic dinner. She has been cooking a lot of healthy meals for us lately, but on Fridays she treats us to an extra bakery item that we try to pace ourselves and enjoy all weekend long – the cookies made it until early afternoon today.

On Saturday Rachel outdid herself on two separate occasions. Before I get to those, however, let me give some background.

We headed into Manhattan for some holiday shopping and sightseeing and had planned to hit up Bloomingdale’s. I must admit, Bloomindale’s was pretty cool and when I have extra stacks of cash lying around the apartment I will probably go back and get some new clothes. After exploring all 9 or 10 stories we’d only made one gift purchase and decided we still had energy for more shopping.

First amazing act: Niketown

As we were walking toward Fifth Avenue we walked right by the big Niketown. There is a huge picture of LeBron James out front and she struck up a conversation with me about LeBron. Anyone who knows Rachel knows that she is not a very big sports fan, but she knows I am, so she humored me. Then she asked, “Do you want to go in?” I was in the market for some new Oregon Ducks gear, so we went in. I told Rachel about the gear I needed and she then struck up a conversation about how good the Ducks are, how much she likes The Little One (her nickname for LaMichal James) and Jeremiah Masoli. Amazing!

We proceeded to spend the next five floors talking about whatever sport we were in the section for. My favorite sections were the English Premier League jersey section (she really wanted me to get some sort of Arsenal Gunner gear) and the Nike Pro Combat Basketball gear. That stuff is intense.

Unfortunately we didn’t find any Oregon gear, but I did have a great time talking sports with Rachel in the middle of our fun shopping outing to Manhattan.

Second amazing act: Oregon vs. Arizona St.

Last night’s game was a late one for us out here on the east coast. It started at 10:20. Rachel has watched the last couple games with me and with each game she has gotten more into them. Before last night’s game she even predicted that the Ducks would put up 67 points. Not too bad of a prediction given how well the offense and how poorly the defense have been playing lately (meaning that the starters would have to stay in and keep scoring 67 points since the defense would keep giving up points). Once the game started – I hope I don’t ruin her reputation as someone who doesn’t like sports – Rachel was more into it than I have ever seen her before. At times she was actually yelling at the TV and LaMichael to break another tackle and score again. I think it was her cheering that was responsible for LaMichael’s three touchdowns in the first twenty minutes of the game – that and his incredible moves.

The game ended around 1:30 am, but Rachel stayed up and watched the entire game with me – and she stayed interested in it the whole time. It was perfect.

I realize Rachel probably hasn’t entirely caught the sports bug, but she has come to terms with me liking sports and has been so wonderful talking to me about them.

Today was great too, by the way. We woke up late after last night’s Duck party and took Crusher for a walk to the park. On the walk we discussed the previous night’s game and where we think the Ducks should be ranked – we both agree the Ducks should be ranked higher than 11, but the two winnable games they lost aren’t helping their cause.

I only hope I can now be as great of a conversationalist to Rachel as she is to me.

NoSQL

Over the last couple months I have been exploring non-relational data stores. It began when I started playing around with Google App Engine and really took hold when I read an interesting paper on Facebook’s open sourced database, Cassandra.

After some experimentation and more reading, I think I like the concept. Non-relational databases are schema-less, which means you can store whatever data you want in each document, and storing your data in documents makes it very easy to throw memcached on top of the datastore.

In trying to find the right database for my needs I tried a few different packages:

Google App Engine Datastore Google gets a lot of credit for starting this movement with their introduction of BigTable. Accessing this datastore through GAE is extremely simple and since it GAE is a hosted service there is nothing to install.

Tokyo Cabinet Tokyo Cabinet is extremely easy to setup and use. TC is not really a server, but rather a library of routines that access a local database file. It didn’t have as many querying options as a few of the others that I encountered so I didn’t do too much with it. If you do need this as a server, there is Tokyo Tyrant.

CouchDB CouchDB was pretty easy to install and fairly easy to use. There is also an excellent admin UI and a pretty cool rest api that allows you to build some impressive apps without much code. Querying is pretty advanced, but it relies on you writing your own MapReduce javascript functions.

MongoDB MongoDB has been my favorite so far. It has some pretty nice querying abilities and some good client libraries. I haven’t built anything big with it yet, but I think this will be my data store of choice.

Screen

I wrote about vi the other day as a tool I am quickly finding I can’t live without. Screen is a development tool that I use everyday and have known for quite a while that I can’t survive without.

From the GNU website, screen is:

Screen is a full-screen window manager that multiplexes a physical terminal between several processes, typically interactive shells.

and a little later:

When screen is called, it creates a single window with a shell in it (or the specified command) and then gets out of your way so that you can use the program as you normally would. Then, at any time, you can create new (full-screen) windows with other programs in them (including more shells), kill the current window, view a list of the active windows, turn output logging on and off, copy text between windows, view the scrollback history, switch between windows, etc.

Essentially that means it is a program that emulates as many terminals as you would like in just one terminal. I love screen because I can set up my full environment and then keep it that way for months or years at a time. Everyday when I get into work, I fire up screen and everything is just where I left it the night before.

Before starting up screen, you need to make sure you have a good ~/.screenrc file. I am not really sure where I stole this one from, but this is the one I use:

vbell on
nethack on 
autodetach on
startup_message off
pow_detach_msg "Screen session of \$LOGNAME \$:cr:\$:nl:ended."
shell -$SHELL
defscrollback 1200
msgminwait 3
hardstatus alwayslastline "%{.cb}%-W%{.rW}%50L> %n %t%{-}%+W%L<%-37=%{+b}%{.bc} %D %Y-%m-%d %{.bY}%c%{-b} %{.bc}%l "

I really am not sure what all those options do, but the one I care most about is the last line. That option places a fixed status line at the bottom of the screen. It numbers all your screens and also provides a few system stats.

Picture 6.png

To fire up screen I use:

screen -D -RR

There are a lot of different options for screen, and I definitely don’t know them all, but these options reattach to my previous screen. Once reattached I am taken back to exactly where I last left off. In the screenshot above I have three screen windows,

  • the first window (#0) I use for both editing swap-bot code and using svn
  • the second window (#1) I use to quickly switch to the database
  • the third window (#2) I use to continuously keep a tail of my apache logs running

Without screen I could either open up three different connections to the remote server and cycle through the windows as needed or just have one window open and cycle through the programs I need by typing the command each time.

To effectively use screen, you only need to know a few commands.

  • <C-a> 0 s takes you to window 0. Any number 0-9 will work.
  • <C-a> <C-a> – takes you back to your last window.
  • <C-a> A – allows you to name the current window
  • <C-a> ‘ – lets you type either the number or name (partial name works too) of the window to switch to
  • <C-a> esc – puts you in the dungeon (not sure why they call it that) and you can scroll back through previous lines using the arrow keys or <C-b> (back) and <C-f> (forward)
  • <C-a> ? – shows you the help where you can learn all the commands

And that is all it takes to become let screen help you become more efficient.

It’s Thursday Again

photo.jpg

It is Thursday night again and this time Crusher offered to buy dinner. Crusher picked up the money I put aside to pay the delivery man and took it to the door. Of course, he would be absolutely terrified if he had to pay the man directly, but he really wanted his JPan!