Detecting hazards from photos with Eevo
In addition to text inputs, Eevo can also process images. This means
you can flexibly ask questions about an image, and get them answered similarly
to any other questions. There is also a special output type box, which can
be used to get the coordinates of a given object within an image.
With its training in understanding consequences, Eevo is especially powerful in tasks that require the model to have a sense of consequences. In this example, we show how to use Eevo to detect hazards from photos taken on construction sites.
Example
In this example, a site supervisor on a restaurant renovation has photographed a trench cut across the dining room floor for a new drain line. Caution tape runs along the far side only, about 6 workers cross the room through the day.

Photo: NPS, public domain.
Below is an example snippet for asking Eevo questions about the photo. The photo is sent in the assets field and referenced by ID in state. Eevo accepts PNG or JPEG photos between 280 and 1,568 pixels.
import base64
import io
import os
import requests
from PIL import Image
API = "https://eevo.io/api/v1"
HEADERS = {"Authorization": f"Bearer {os.environ['EEVO_API_KEY']}"}
def as_asset(path):
"""A photo as a SystemOne image asset, shrunk to fit Eevo's 1,568-pixel limit."""
image = Image.open(path).convert("RGB")
image.thumbnail((1568, 1568))
buffer = io.BytesIO()
image.save(buffer, format="JPEG", quality=90)
return {"type": "image", "format": "jpg", "data_base64": base64.b64encode(buffer.getvalue()).decode()}
QUESTIONS = {
"hazard": {
"type": "noul",
"instructions": "The photo shows a hazard that could injure someone working here.",
},
"hazard_type": {
"type": "choice",
"instructions": "What is the main hazard in the photo?",
"criteria": {
"opening": "An open trench, hole or edge someone could step or fall into",
"falling_object": "Something that could fall on someone",
"vehicle": "Vehicles or moving machinery that could strike someone",
"fall_from_height": "Work at height without protection",
"slip_trip": "Something on the floor that could make someone slip or trip",
"electrical": "Exposed or damaged electrical equipment",
"none": "No hazard",
},
},
"outcome": {
"type": "score",
"instructions": "If nothing changes, what is the most likely outcome for the people working here?",
"criteria": [
"No harm",
"A near miss",
"A minor injury treated with first aid",
"A serious injury with time off work",
"A fatality or several serious injuries",
],
},
"when": {
"type": "choice",
"instructions": "If it causes harm, how soon is that likely?",
"criteria": {
"hours": "Within hours of normal work",
"days": "Within days of normal work",
"weeks": "Over weeks or months",
},
},
"where": {"type": "box", "instructions": "the part of the photo most likely to injure someone"},
}
response = requests.post(
f"{API}/systemone",
headers=HEADERS,
json={
"model": "eevo-1",
"state": "Morning safety walk, restaurant renovation, dining room. About 6 workers cross this room "
"with materials all day. Photo: dining_room",
"questions": QUESTIONS,
"assets": {"dining_room": as_asset("renovation-trench.jpg")},
},
)
response.raise_for_status()
answers = response.json()["answers"]Below is an example response:
{
"hazard": { "type": "noul", "noul": 0.8978 },
"hazard_type": { "type": "choice", "choice": "opening", "confidence": 0.9673, "probabilities": { "opening": 0.9673, "falling_object": 0.0095, "vehicle": 0.0044, "fall_from_height": 0.0034, "slip_trip": 0.0101, "electrical": 0.0025, "none": 0.0029 } },
"outcome": { "type": "score", "score": 2.1335, "confidence": 0.3132, "legend": { "0": "No harm", "1": "A near miss", "2": "A minor injury treated with first aid", "3": "A serious injury with time off work", "4": "A fatality or several serious injuries" }, "probabilities": { "0": 0.1099, "1": 0.2201, "2": 0.2266, "3": 0.3132, "4": 0.1301 } },
"when": { "type": "choice", "choice": "hours", "confidence": 0.6036, "probabilities": { "hours": 0.6036, "days": 0.2627, "weeks": 0.1337 } },
"where": { "type": "box", "box": { "top": 0.6756, "bottom": 0.9222, "left": 0.1006, "right": 0.9144 }, "confidence": 0.5486 }
}As usual, the answers come with probabilities, which can be used to set thresholds, such as only logging photos where a hazard is unlikely and passing the rest on for a decision. The box is given as fractions of the image's height and width, and can be drawn on the photo:
from PIL import ImageDraw
image = Image.open("renovation-trench.jpg")
box = answers["where"]["box"]
w, h = image.size
ImageDraw.Draw(image).rectangle(
[box["left"] * w, box["top"] * h, box["right"] * w, box["bottom"] * h], outline="red", width=6
)
image.save("renovation-trench-marked.jpg")
Bonus: Using Consequence-based decisions API to act on the detected hazard
Below is an example of passing the photo answers, with additional context, to the Consequence-based Decision API. There are 3 options: stop now and fit barriers, tape the edge and fit barriers at lunch, or leave the trench as it is until it is backfilled.
a = answers
context = f"""A building contractor's site supervisor is deciding what to do about a hazard found on the morning
safety walk of a restaurant renovation. The photo shows a trench about 50 cm deep cut across the concrete floor of
the dining room for a new drain line. Caution tape runs along the far side only; the near edge, where workers cross
with materials, is open. About 6 workers cross the room through the day. The trench is backfilled in 2 days.
An image check of the photo gave: hazard present with probability {a['hazard']['noul']:.2f}; hazard type
{a['hazard_type']['choice']} ({a['hazard_type']['confidence']:.2f}); if nothing changes, most likely outcome score
{a['outcome']['score']:.1f} on a 0-4 scale from no harm to fatality; expected to matter within {a['when']['choice']}
({a['when']['confidence']:.2f}).
Left as it is for 2 days, the supervisor estimates about a 10% chance someone steps or falls into the trench and,
if they do, a 30% chance of a lost-time injury such as a fracture: about a 3% chance in all. One costs the contractor
about €40,000. Rigid barriers and a plank bridge are in the van: fitting them takes the crew about 45 minutes.
Extra tape takes 5 minutes, but people step over tape."""
def consequence(stakeholder, valence, label, description):
return {"stakeholders": [stakeholder], "valence": valence, "label": label, "description": description}
OPTIONS = [
{"id": "A", "description": "Stop work in the dining room now and fit rigid barriers along both edges and a plank bridge (about 45 minutes).",
"consequences": [
consequence("workers", "benefit", "Fall risk removed now",
"The about 3% chance that a worker suffers a lost-time injury (such as a fracture) by falling into the trench over the next 2 days is removed."),
consequence("workers", "harm", "45 minutes of fitting barriers",
"The crew spends about 45 minutes fitting barriers instead of their planned work, and their day runs a little later."),
consequence("contractor", "benefit", "Injury cost avoided",
"About €1,200 of expected injury costs avoided (a 3% chance of a lost-time injury costing about €40,000)."),
consequence("contractor", "harm", "Crew time", "About €300 of crew time (6 workers for about 45 minutes)."),
]},
{"id": "B", "description": "Tape the near edge now (5 minutes), keep working, and fit the barriers and bridge at lunch in about 3 hours.",
"consequences": [
consequence("workers", "benefit", "Most fall risk removed",
"The chance of a lost-time injury falls from about 3% to about 0.3% (3 more hours with only tape, which people step over)."),
consequence("workers", "harm", "Barriers fitted in the lunch break",
"The crew fits the barriers over about 45 minutes around lunch, cutting into their break."),
consequence("contractor", "benefit", "Most injury cost avoided",
"About €1,080 of expected injury costs avoided (the chance of a lost-time injury costing about €40,000 falls by about 2.7 percentage points)."),
consequence("contractor", "harm", "Crew time",
"About €320 of crew time (5 minutes of taping now and about 45 minutes at lunch)."),
]},
{"id": "C", "description": "Leave it as it is until the trench is backfilled in 2 days.",
"consequences": [
consequence("workers", "harm", "Fall risk stays for 2 days",
"Workers face about a 3% chance of a lost-time injury from falling into the trench over the next 2 days."),
consequence("contractor", "harm", "Expected injury costs",
"About €1,200 of expected injury costs (a 3% chance of a lost-time injury costing about €40,000)."),
consequence("contractor", "benefit", "No crew time spent",
"No crew time spent on barriers, saving about €300 compared with fitting them."),
]},
]
decision = requests.post(
f"{API}/cb-decision",
headers=HEADERS,
json={
"model": "eevo-1",
"mode": "default",
"context": context,
"options": OPTIONS,
"stakeholders": [
{"id": "workers", "description": "The workers on the renovation",
"values": "Going home uninjured; a workable day."},
{"id": "contractor", "description": "The building contractor",
"values": "Safe work, schedule and cost."},
],
},
)
decision.raise_for_status()
for result in decision.json()["results"]:
print(result["stakeholder"]["id"], result["best"], result["probabilities"])In this example, both stakeholders are best served by stopping now to fit the barriers.
Caveat around using the box type
The box question type always produces a "most likely" answer, meaning it cannot respond "not found". Therefore, one should usually couple the box questions with other questions that ask whether something exists in the photo or not.
