A data collection and visualization application for domestic-violence (DV) incidents involving technology. It scrapes police logs and court decisions, stores them in a MySQL database, and serves a Flask web UI for keyword search, tagging, and time-series charts (incident counts and percentages over time).

The project was built to help answer: what kinds of crimes are happening in the local community that are associated with abuse using technology?

What it tracks

Two entity types, both taggable with case and device tags:

• PoliceLog — police blotter entries (primarily Cambridge, MA). Fields include report time, location, content, incident type, source, and incident number.
• Decision — court opinions. Fields include year, month, title, content, place, and source.

Tags are free-form with two categories: case (e.g. "harassment by technology", "domestic dispute") and device (e.g. "iPhone", "PC"). Suggested values are defined in frontend/consts.py.

Repository structure

frontend/          Flask web application (the main product)
  main.py            All HTTP routes (search, detail, charts, tag editing, reporting)
  db.py              Peewee ORM models + MySQL connection
  consts.py          Hardcoded lists of suggested incident types and devices
  flask_login.py     Vendored flask-login 0.2.10 (bundled, not pip-installed)
  flask_pagination.py Vendored flask-paginate (bundled, not pip-installed)
  index2.fcgi        FastCGI entry point for production (via flup)
  ipvtech.sql        MySQL schema + seed data dump (Navicat export)
  static/            Bootstrap 3, jQuery, d3 v3, nv.d3, bootstrap-tagsinput
  task_scripts/      Scraping and data-ingest scripts (run separately from the web app)
crawler/           Standalone Cambridge police-log scraper (SQLite-backed)
images/            Project logos

Requirements

This is a Python 2 codebase. There is no requirements.txt; install the following under a Python 2 interpreter:

• Flask
• peewee (ORM)
• requests
• BeautifulSoup4 (bs4)
• MySQLdb (mysql-python)
• flup (for FastCGI in production)
• CasperJS (only for the court-decisions scraper in task_scripts/)

The MySQL database ipvtech must be reachable on localhost as root with an empty password (hardcoded in db.py).

Setup

1. Create the database and load the schema:

   mysql -u root ipvtech < frontend/ipvtech.sql

1. Install the Python dependencies above under Python 2.

2. The Flask templates (index.html, result.html, decision_result.html, case.html, decision.html, report.html, login.html) are referenced by main.py but are not present in this repository. They must be supplied in a frontend/templates/ directory before the app can render pages.

Running

Development (from the frontend/ directory):

python main.py

This starts Flask's built-in server with debug mode on.

Production (FastCGI via flup):

python frontend/index2.fcgi

Data collection

Police logs and court decisions are gathered by separate scripts that run independently of the web app:

• Cambridge police logs — crawler/police_log_scraper.py (writes to local SQLite) or frontend/task_scripts/scrape_cambridge_police_log.py (writes directly to MySQL).
• Massachusetts court decisions — frontend/task_scripts/scrape_mass_court_reports.js (CasperJS) produces JSON, which is then loaded into MySQL by frontend/task_scripts/store_mass_count_reports.py.

License

BSD 2-Clause style license. See frontend/LICENSE.md, crawler/LICENSE, and images/LICENSE.md for the full text. Copyright (c) 2015–2017 Andrew Lewman and Sarah Cortes.

Contributing

See AGENTS.md for detailed contributor guidance: architecture, code conventions, non-obvious patterns, and known gotchas.