Data Scraping with Cypress

I used Cypress and Node.js to gather data on my local city councillors and created a simple web app to find your representative by postcode.
Screenshot of the city councillors app
Find your councillor by postcode lookup or by locating your ward on the map.

I built this prototype because I wanted something to include in my portfolio that would demonstrate my ability to work on consumer-facing, mobile-friendly web apps with an eye for design and an understanding of interactivity on touch devices. I decided to work on a small app for residents of my city to find their ward councillors. This would give me an opportunity to work with a common mobile design pattern — a fixed viewport rather than a scrollable document — and experiment with mapping tools and automated data collection.

I found a GeoJSON file that described the city's ward boundaries from Edinburgh's open data portal. The ward councillors were available from edinburgh.gov.uk, but not in a machine-readable format. I wrote a Cypress script (source), which I often use for integration testing, to scrape data from the site. The script visits the government site and collects every councillor's name, phone number, email address, photo, and website into a JSON array.

Snippet from councillors.json
[
  {
    "name": "Almond",
    "number": 1,
    "councillors": [
      {
        "url": "https://democracy.edinburgh.gov.uk/mgUserInfo.aspx?UID=115",
        "name": "Kevin Lang",
        "phone": "01315294389",
        "email": "kevin.lang@edinburgh.gov.uk",
        "party": "Scottish Liberal Democrats",
        "photo": "images/kevin-lang.jpg"
      }
    ]
  }
]

Unfortunately, the photos on the government website were all different sizes and aspect ratios. To fit them into my design, I wrote a quick Node.js script (source) to crop the photos so that they were all square.

Diagram of image changes after cropping
The source photos were tiny! 🙄 After adjusting the aspect ratio, I left them at different sizes to retain every pixel I could.

I then built the web app using Vue and Vite. I wanted a UI that would be familiar to someone using a mobile device, so I mimicked the vertical layout of Google maps on my phone. But this layout doesn't work well in landscape orientation — when the viewport is wider than it is tall.

Screenshot in landscape without orientation media query
The map is almost entirely covered on a phone being held sideways.

Traditional breakpoints, based on device width, weren't a great option. A screen width of 767px might represent a tablet in portrait or a phone in landscape. Rather than use a lot of media queries at set widths, I used orientation media queries to apply a different layout for viewports in portrait or landscape orientation.

Example of using orientation media queries
.app {
  /*
   * Use default "flow" layout for
   * top-to-bottom display when the
   * viewport is taller than it is
   * wide.
   */
}

@media (orientation: landscape) {

  .app {
    display: grid;
    grid-template-columns: var(--sidebar-min-width) auto;
  }
}

This reduced the amount of CSS I had to write to account for different screen sizes, since the UI scaled pretty well once it used the appropriate layout for each orientation.

Screenshot in landscape with orientation media query
It's easy to see the map and the councillors now.

I used the popular open source mapping tool Leaflet, along with tiles from OpenStreetMaps, to render the map with the ward boundaries. There's a well-mainted Vue component (vue-leaflet) that I could have used, but I wanted to work directly with the underlying library. This way, I could use what I learned regardless of which framework or toolset I might work with in the future.

The prototype is really small, so I wrote all of the map interactions in the root component. Wards are highlighted, postcodes pinpointed, and the map is zoomed all from the same single-file-component. In a production setting, I'd expect to separate the map controller logic from the app layout component, unless I was confident the app's complexity wasn't going to grow over time.

Finally, I added integration tests to cover all the functionality. The tests are all pretty simple with this app, but I have a lot of experience working with Cypress in my work with the Public Knowledge Project. If you're interested, you can see a sample commit or view all of my commits to the test suites for OJS, OMP, and OPS.