Pivotal Labs

Main menu

Skip to primary content
Skip to secondary content
  • About
  • Case Studies
  • Team
    • Executives
    • Locations
      • San Francisco (HQ)
      • Boston
      • Boulder
      • Denver
      • London
      • Los Angeles
      • New York
  • Community
    • Blogs
    • Tech Talks
    • Events
  • Careers
    • Lifestyle
    • Principles & Practices
    • Benefits
    • FAQ
    • Apply
  • Contact
    • Press Room
    • Press Releases
    • In The News
    • Press Kit
  • All
  • Labs
  • Standup
  • Tracker

Access your database’s best features with Sequel

Brandon Liu
Monday, May 6, 2013

Sequel is a wonderful library for interacting with relational databases. Some of my favorite aspects:
1. Out-of-the box support for foreign key constraints.
2. Straightforward migration DSL.
3. Support for all major free RDBMSes and even some proprietary ones. Also JRuby support through JDBC. (although JDBC doesn’t necessarily
give you all the features a native driver would. Projects like jruby-pg are making progress though.)
4. “One way to do it” for common operations such as inserts and updates. Parsimonious API (compared to ActiveRecord).
5. bin/sequel executable gives you an interactive ruby session directly from a database URL.
6. Extremely powerful Dataset abstraction lets you write general query code.
7. The source code itself is a pleasure to work with and the maintainer (jeremyevans) is extremely diligent about pull requests.

One feature i’d like to expand upon is the ability to call raw SQL functions from Sequel’s DSL.

Here’s a migration where I create a speeches table with some full-text search columns:


Sequel.migration do
  up do
    create_table :speeches do
      primary_key :id
      String :text
      String :speaker
      Integer :year
    end

    run "ALTER TABLE speeches ADD COLUMN ts_text tsvector;"
    run "CREATE INDEX ts_text_idx ON speeches USING gin(ts_text);"
    run "CREATE TRIGGER tsvectorupdate BEFORE INSERT OR UPDATE
         ON speeches FOR EACH ROW EXECUTE PROCEDURE
         tsvector_update_trigger(ts_text, 'pg_catalog.english', text, speaker);"
  end
end

 

And here’s some sample code to query it:


require 'sequel'

def search(dataset, term)
  results = dataset.select do
    [ts_headline('english',
                 :text,
                 to_tsquery('english', term),
                 'MaxFragments=2').as(headline),
     id,
     speaker]
  end
  results.filter("ts_text @@ to_tsquery('english', ?::text)", term)
end

DB = Sequel.connect('postgres://localhost/blogpost')

DB[:speeches].insert(:year => 1865, :speaker => 'President Lincoln', :text => 'Fourscore and seven years ago')
DB[:speeches].insert(:year => 1963, :speaker => 'President Kennedy', :text => 'I am a Jelly Donut')

p search(DB[:speeches], 'President').all # returns both speeches
p search(DB[:speeches].where("year > ?", 1900), 'President').all # returns only the Kennedy speech

The search function takes in a dataset and a query term, and returns a new dataset filtered with a body or speaker name matching the query term,
as well as a “headline”, or the context around the search term to display in search results. This takes full advantage of a full-text search index.

Sequel is also great as glue code for calling PostGIS functions. I’ve found it especially useful for DRY-ing up long sets of raw SQL queries into maintainable scripts.

 

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Python web testing for the Ruby programmer

Brandon Liu
Monday, April 29, 2013

Ruby has a couple of well-known libraries for unit testing, mocking and stubbing HTTP interactions. My typical toolset includes RSpec, WebMock and VCR. I had the chance to work on a Python project recently and did some investigation into similar libraries for Python.

General testing libraries

The two most popular python test libraries + runners are py.test and nose. A neat feature of py.test is that is gets by with only the builtin ‘assert’ keyword: the test runner expands assertion failures to show exactly how the assertion failed e.g.:

===================== FAILURES =====================
_____________________ test_web _____________________

    def test_web():
      resp = c.get('/')
>     assert resp.status == '404'
E     assert '200 OK' == '404'
E       - 200 OK
E       + 404

lib/test_snack_overflow.py:9: AssertionError
============= 1 failed in 0.24 seconds =============

One gotcha is that py.test silences stdout by default, so you’ll need to pass the -s flag to see any debugging output.

Test doubles

The mock library can be installed through pip for Python 2 and comes as part of the standard library of Python 3. As an example of mocking out an environment variable using a context manager:

with mock.patch.dict('os.environ', {'APP_NAME': 'HelloWorld'}):
assert app.name = 'HelloWorld'

There’s also a decorator form, nice because it clearly states what modules are being mocked in your tests:

@mock.patch('crazyservice')
def test_nationals_stats(_crazyservice):
_crazyservice.date = "2013-04-06 12:31 AM"

Another gotcha for newcomers to Python testing: not all methods can be replaced at runtime. A good example is stubbing DateTime.now in ruby: the equivalent monkeypatching in python raises:

TypeError: can’t set attributes of built-in/extension type ‘datetime.datetime’

as these methods cannot be replaced in CPython. You’ll need to write a wrapper function and mock that out instead in your tests.

Web testing

The werkzeug WSGI library includes simple utilities to make request testing as easy as with rack-test. Another library I’m a big fan of is HTTPretty, which is similar to Ruby’s WebMock or FakeWeb. Finally, Ruby’s VCR gem is invaluable for recording real HTTP interactions. A port exists on python called vcr.py.

 

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

[SF] Standup 1/23

Brandon Liu
Wednesday, January 23, 2013

Interestings

Unwrapping an annotated tag in Git

Annotated tags are their own "objects" and the sha does not point to the commit that they are tagging. In order to unwrap the tag, add ^{commit} to the original sha. This will not harm regular shas, so you can use it in scripts where you may encounter annotated tags.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

[SF] Standup 1/22

Brandon Liu
Tuesday, January 22, 2013

Helps

How do I get sprockets to output its dependency graph?

Answer:

Call "dependency" method on an asset object.

What is first step in debugging remote irb that won't startup because of missing "readline" dependency?

Your ruby was compiled and then the readline library was deleted or moved. You probably need to recompile ruby. Likely that you have readline already, otherwise nothing would work.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

New Jasmine Gem

Brandon Liu
Friday, August 3, 2012

Helps

  • New Jasmine Gem

There’s a new version of jasmine-gem on GitHub – please test it out so we can push a new gem next week.

https://github.com/pivotal/jasmine-gem

  • jQuery opacity

Once the CSS opacity is set on an element with jQuery the z-index is no longer respected and the element appears on the bottom.

Leads:
Can it be reproduced in other browsers? Only tried Chrome so far.

Interestings

  • Safari doesn’t like Rails timestamps

my_model.created_at or my_model.create_at.to_i spit out values that when you (in JavaScript) new Date(value), give crazy dates in Safari.

The workaround is to use:

(Ruby)
value = my_model.created_at.to_f * 1000

(JS)
date = new Date(value)

Discussion:
Try all browsers when dealing with a Date API
There’s also DateJS, but has not been maintained in a long time.

  • Jasmine failed to start on latest Chrome

If you have an updated version of Chrome and jasmine fails to start Chrome, reinstalling later version of chrome driver (http://code.google.com/p/chromedriver/downloads/list).

Or alternative is to re-image the machine, which contains later Chrome and later ChromeDriver.

  • bitbucket Tracker integration broken by ‘&’ in author name

We spent several hours tracking down this (open) bug in bitbucket’s tracker broker. You can change the ampersand to something else in your git-pair script.

  • IE9 svg path click handler

IE9′s click handler for SVG paths treats two clicks as a single click and six clicks as a double click.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

Like Dreamcatcher, but for Shapes

Brandon Liu
Wednesday, August 1, 2012

Helps

  • capybara-webkit save_and_open_page returns empty DOM after AJAX request

Investigate:
Use page.driver.render to save a screenshot.
Look at capybara issue discussed yesterday at http://pivotallabs.com/users/abruce/blog/articles/2220-rager-party-standup-7-31-12-

Interestings

  • Shapecatcher: draw something, find the closest-matching Unicode character

http://shapecatcher.com/

Also of note: http://nice-entity.com/

  • bundle exec foreman

Using bundle exec may solve some Foreman issues teams have been seeing.

  • 0 Shares
  • Share on Facebook
  • Share on Twitter

SF Standup 7/30

Brandon Liu
Monday, July 30, 2012

Helps

  • Apache proxy: (502)Unknown error 502: proxy: pass request body failed

We need to make requests to a server and these requests have to come from a fixed IP address and be authenticated via SSL. For this purpose we set up a EC2 instance to function as a proxy. We can make cURL requests using our key and certificate from the proxy machine successfully. But when we go through Apache we get:
“(502)Unknown error 502: proxy: pass request body failed”

Interestings

  • Backbone’s views’ $el contains all matched elements

If you initialize a Backbone view with an el selector that matches more than one element: within the view $el will refer to a jQuery collection that contains all the matched elements, while el will contain the DOM object for only the first element matched. To achieve consistency, you might want to instantiate your view like so:

new view({el: '.selector_that_matches_many:eq(0)'});
  • 0 Shares
  • Share on Facebook
  • Share on Twitter
Brandon Liu

Brandon Liu
San Francisco

Subscribe to Brandon's Feed

Author Topics

bloggerdome (2)
postgres (1)
python (1)
agile (3)
  • About
  • Case Studies
  • Team
  • Community
  • Careers
  • Contact
  • Labs
  • Events

Contact Us

contact@pivotallabs.com
+1 415-77-PIVOT
TwitterLinkedInFacebook

Pivotal Tracker

Tracker is the award-winning agile project management tool that enables real-time collaboration around a shared, prioritized backlog.
Visit pivotaltracker.com >