Guidance for agents working in this repository.

What this project is

IPVTechDVData: a domestic-violence (DV) data collection app. It scrapes police logs and court decisions, stores them, and serves a Flask web UI for keyword search, tagging, and time-series charts (count + percentage over time). Two entity types are tracked: PoliceLog (police blotter entries) and Decision (court opinions). Both share a common Tag system with two tag types: case and device.

Version control: Fossil, not Git

This is a Fossil repository (.fslckout), not Git. The repository file is ../IPVTech.fossil. Use fossil commands, e.g. fossil status, fossil diff, fossil commit. There is no .git.

Python 2 codebase

All Python code targets Python 2 and will not run on Python 3 without porting:

• print statements (not print(...) calls)
• import md5 and md5.new(...).hexdigest() (the md5 module was removed in Py3)
• from urlparse import ... / urllib style imports
• except sqlite3.ProgrammingError: (this exception doesn't exist in the way used here)

There is no requirements.txt, setup.py, or Makefile. Dependencies must be installed manually under a Python 2 interpreter:

• Flask
• peewee (ORM)
• requests
• BeautifulSoup (bs4)
• MySQLdb (mysql-python)
• flup (for FastCGI)

Repository layout

frontend/          Flask web application (the main product)
  main.py            App entry point, all routes
  db.py              Peewee ORM models + MySQL connection
  consts.py          Hardcoded incident type/device lists for tag suggestions
  flask_login.py     VENDORED copy of flask-login 0.2.10 (see below)
  flask_pagination.py VENDORED copy of flask-paginate (see below)
  index2.fcgi        FastCGI entry point for production (flup)
  ipvtech.sql        MySQL schema + seed data dump (Navicat export, ~2014)
  static/            Bootstrap 3, jQuery, d3 v3, nv.d3, bootstrap-tagsinput
  task_scripts/      Scraping/ingest scripts (run separately from the web app)
crawler/           Standalone Cambridge police-log scraper (sqlite-based)
images/            Project logos

Vendored libraries — do NOT pip install these

frontend/flask_login.py and frontend/flask_pagination.py are vendored copies committed into the repo. main.py imports them as local modules (from flask_login import LoginManager, from flask_pagination import Pagination). If you pip install flask-login it will shadow/conflict with these. Treat them as part of the codebase, not external dependencies. Do not assume they match any particular upstream release behavior.

Missing templates directory

main.py calls render_template() for: index.html, result.html, decision_result.html, case.html, decision.html, report.html, login.html. No templates/ directory exists in the repo. The app cannot render any page as-is. Either the templates were never committed or were removed. This is the single biggest blocker to running the frontend locally.

Two parallel data pipelines

There are two separate, divergent implementations of the Cambridge scraper — be careful which one you're looking at:

1. crawler/police_log_scraper.py — writes to a local SQLite file policelog.sqlite (schema in crawler/policelog_db.sql). Columns: report_date, report_time, occur_date, occur_time as separate TEXT fields.
2. frontend/task_scripts/scrape_cambridge_police_log.py — near-identical logic but writes to MySQL ipvtech.policelog via MySQLdb, using string-interpolated SQL (SQL-injection-prone). Same separate date/time column layout.

The actual MySQL schema in ipvtech.sql (used by the Flask app) instead has combined report_time and occur_time DATETIME columns — no report_date/occur_date columns. So the scraper column layout does not match the schema the web app expects. This schema drift is pre-existing and unresolved.

The court-decisions pipeline is separate:

• task_scripts/scrape_mass_court_reports.js — CasperJS (PhantomJS-based) script that scrapes massreports.com into result_list.json / result_detail.json. Requires CasperJS installed, not Python.
• task_scripts/store_mass_count_reports.py — loads the JSON into MySQL decision table and back-fills month by regex-matching month names in the content.

Database

• Frontend ORM: Peewee, connecting to MySQL ipvtech as root with empty password on localhost (hardcoded in db.py). db.py calls mysql_db.connect() at import time, so importing the module opens a DB connection.
• Models (db.py): PoliceLog, Decision, Tag, PoliceLogTag, DecisionTag, User.
• User model implements the flask-login auth interface (is_authenticated, is_active, is_anonymous) but lacks get_id/id-retrieval matching the vendored flask-login's expectations; load_user in main.py uses User.get(User.id==userid).
• ForeignKey redundancy: PoliceLogTag and DecisionTag declare both an explicit IntegerField (policeLog_id/decision_id) and a ForeignKeyField. Peewee auto-creates a _id column from the FK, so there are overlapping column definitions. Note DecisionTag declares decision_id before the decision ForeignKeyField.
• ipvtech.sql is a Navicat dump from ~2014 with seed rows; the policelog table is MyISAM with AUTO_INCREMENT=25849.

Security-sensitive hardcoded values (pre-existing, insecure)

These are committed in source — do not propagate the pattern, and do not assume they are secret:

• app.secret_key = 'the manipulative man' (main.py)
• md5_salt = 'coersive control' (note the misspelling of "coercive") — passwords stored as md5(salt + password) (main.py, db.py)
• MySQL root with empty password (db.py, task_scripts/*.py)
• task_scripts/scrape_cambridge_police_log.py builds SQL via % string formatting (SQL injection risk).

Running the app

• Dev: python main.py (from frontend/) — calls app.run(debug=True).
• Prod: index2.fcgi served via flup FastCGI.
• Both require a reachable MySQL ipvtech database populated from ipvtech.sql, the vendored libs importable (i.e. run from the frontend/ dir), and — currently missing — a templates/ directory.

Routes & data flow (main.py)

Route	Method	Purpose
/	GET	Index page
/search	GET	Paginated PoliceLog search (keyword/source/tag filters)
/decisions	GET	Paginated Decision search (same filter pattern)
/search_data	GET	JSON time-series (count + %) for PoliceLog, consumed by nv.d3 charts
/decision_data	GET	JSON time-series for Decisions
/case/<id>	GET	Single PoliceLog detail + tags
/decision/<id>	GET	Single Decision detail + tags
/report	GET	Self-report incident form
/insert	POST	Insert a user-reported PoliceLog (source="reported")
/register	POST	Register user (md5 password), logs them in
/login	POST	Stub — returns "hello", not implemented
/save_tags, /save_decision_tags	GET	Replace all tags on a record (case + device), via query-string type_tags/device_tags

Conventions to know when editing main.py

• Peewee ** operator = SQL LIKE (e.g. PoliceLog.location ** sql_key); >> None = IS NULL.
• Keyword search builds sql_key = "%" + keyword + "%".
• fn.Date_format, fn.Concat, fn.Year, fn.Month are MySQL-specific Peewee helpers — the app is MySQL-only, not DB-portable.
• Tag-saving routes delete all existing tags then re-insert (no diffing), and split comma-separated tag strings, lowercasing them. New tag names are created on demand.
• consts.incident_types / consts.incident_devices are passed to templates as suggestion lists for the tag input (bootstrap-tagsinput).
• records_per_page = 10, pagination via the vendored Pagination(bs_version=3).
• The time-series endpoints (*_data) iterate month-by-month from min to max year/month, filling gaps with zeros, and return both absolute counts and a percentage-of-base series.

Gotchas summary

• No tests exist anywhere in the repo.
• crawler/ and frontend/task_scripts/scrape_cambridge_police_log.py are duplicate scrapers with different backends (SQLite vs MySQL) — changes to scraping logic need to be applied in both places.
• flask_login.py imports from flask.signals import Namespace, which was removed in modern Flask — another reason the code only runs on old Flask under Python 2.
• A committed court_decisions.db SQLite binary lives in task_scripts/ (leftover from an earlier SQLite-based approach to storing decisions).
• frontend/README.md is a one-liner; crawler/docs/Schedule and Time Tracking.txt contains the original project planning notes, member list, and requirements (useful historical context).