SmartSpectra Metrics Payload Data Types

Origin of an Insight. Only INSIGHT_TYPE_VITALS is emitted on delivered Insights today; SPEECH and COMBINED are reserved for a future release. Correlate an on-demand reply with its prompt via Insight.request_id, not this field.

enum InsightType {
  INSIGHT_TYPE_VITALS = 0;
  INSIGHT_TYPE_SPEECH = 1;
  INSIGHT_TYPE_COMBINED = 2;
}

• INSIGHT_TYPE_VITALS -- Auto-fired vitals snapshot dispatched periodically by InsightSession with the accumulated metrics buffer and no caller prompt. The only value set on delivered Insights today.
• INSIGHT_TYPE_SPEECH -- Reserved, not emitted. Describes a request from RequestInsight with a prompt but no buffered metrics at dispatch time (prompt-only nudge).
• INSIGHT_TYPE_COMBINED -- Reserved, not emitted. Describes a request from RequestInsight carrying both the caller's prompt and a non-empty metrics buffer.

LLM-generated analysis or error returned by the insights endpoint.

message Insight {
  int32 request_id = 1;
  string processed_at = 2;
  oneof result {
    string analysis = 3;
    string error = 4;
  }
  InsightType type = 5;
}

• int32 request_id -- Client-generated request ID. Populated client-side by the async callback closure (not returned by the server). Lets callers correlate responses to requests.
• string processed_at -- ISO timestamp of when the analysis was produced
• string analysis -- The LLM analysis text, set on success.
• string error -- Error message, set on failure.
• InsightType type -- Origin of this insight. Always INSIGHT_TYPE_VITALS today — SPEECH and COMBINED are reserved and never set — so route on request_id rather than on this field to tell auto-fired snapshots from on-demand replies.

FeatureType defines high-level physiological measurement categories. Each feature represents a super-metric that may encompass multiple individual metrics.

enum FeatureType {
  BREATHING = 0;
  EDA = 2;
  FACE = 3;
  CARDIO = 4;
}

Some wire values are intentionally omitted from this view; the gaps in the numeric sequence preserve compatibility with the underlying proto. Use the listed names — do not renumber.

• BREATHING -- Breathing measurements (chest and abdomen)
• EDA -- Electrodermal activity (EDA) measurement
• FACE -- Facial tracking and analysis
• CARDIO -- Cardiovascular metrics (pulse, HRV, blood pressure)

MetricType defines individual physiological measurements that can be requested and produced during video-based vital signs analysis.

enum MetricType {
  CHEST_BREATHING = 0;
  ABDOMEN_BREATHING = 1;
  BREATHING_RATE = 2;
  BREATHING_AMPLITUDE = 3;
  APNEA = 4;
  RESPIRATORY_LINE_LENGTH = 5;
  BASELINE = 6;
  INHALE_EXHALE_RATIO = 7;
  EDA_TRACE = 10;
  FACE_LANDMARKS = 11;
  BLINKING = 12;
  TALKING = 13;
  EXPRESSIONS = 14;
  PULSE_RATE = 15;
  ARTERIAL_PRESSURE_TRACE = 16;
  HRV = 17;
}

Some wire values are intentionally omitted from this view; the gaps in the numeric sequence preserve compatibility with the underlying proto. Use the listed names — do not renumber.

• CHEST_BREATHING -- Breathing upper (chest) metrics
• ABDOMEN_BREATHING -- Breathing lower (abdomen) metrics
• BREATHING_RATE -- Breathing aggregate metrics
• EDA_TRACE -- EDA metrics
• FACE_LANDMARKS -- Face metrics
• PULSE_RATE -- Cardio metrics

Wrapper message for passing a list of requested metrics through MediaPipe packets

message RequestedMetrics {
  repeated MetricType metrics = 1;
}

Represents a single measurement with timestamp and stability information. Used for various physiological measurements throughout the system.

message Measurement {
  float value = 1;
  bool stable = 2;
  int64 timestamp = 3;
}

• float value -- The measured or estimated value
• bool stable -- Whether the measurement is considered stable/reliable
• int64 timestamp -- Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch

Represents detection status with timestamp information. Used to track whether a particular physiological feature or state is detected.

message DetectionStatus {
  bool detected = 1;
  bool stable = 2;
  int64 timestamp = 3;
}

• bool detected -- Whether the feature/state was detected
• bool stable -- Whether the detection is considered stable/reliable
• int64 timestamp -- Absolute timestamp at which the detection status was updated, in microseconds, since Linux epoch

Represents a measurement with an associated confidence score. Extends basic measurement with confidence information for quality assessment.

message MeasurementWithConfidence {
  float value = 1;
  bool stable = 2;
  float confidence = 3;
  int64 timestamp = 4;
}

• float value -- The measured value
• bool stable -- Whether the measurement is considered stable/reliable
• float confidence -- Confidence score for the measurement, expressed as a percentage in the range [0.0, 100.0]
• int64 timestamp -- Absolute timestamp at which the measurement was taken, in microseconds, since Linux epoch

Enumerates the supported facial expression types output by the model.

enum ExpressionType {
  UNSPECIFIED = 0;
  ANGRY = 1;
  CONTEMPT = 2;
  DISGUST = 3;
  FEAR = 4;
  HAPPY = 5;
  NEUTRAL = 6;
  SAD = 7;
  SURPRISE = 8;
}

• UNSPECIFIED -- Expression is unspecified or unknown
• ANGRY -- Angry expression
• CONTEMPT -- Contempt expression
• DISGUST -- Disgust expression
• FEAR -- Fear expression
• HAPPY -- Happy expression
• NEUTRAL -- Neutral expression
• SAD -- Sad expression
• SURPRISE -- Surprise expression

Associates an expression type with its confidence score.

message ExpressionScore {
  ExpressionType type = 1;
  float confidence = 2;
}

• ExpressionType type -- Expression type identifier
• float confidence -- Confidence score for the expression, expressed as a percentage in the range [0.0, 100.0]

Represents a detected expression with metadata. Used for facial expression analysis and emotion detection.

message Expression {
  bool stable = 1;
  int64 timestamp = 2;
  repeated ExpressionScore scores = 3;
}

• bool stable -- Whether the detection is considered stable/reliable
• int64 timestamp -- Absolute timestamp at which the expression was detected, in microseconds, since Linux epoch
• repeated ExpressionScore scores -- Confidence distribution across all expression types

Represents a single HRV measurement computed over some period of pulse data.

message Hrv {
  double rmssd = 1;
  double mean_nn = 2;
  double sdnn = 3;
  double baevsky = 4;
  int64 timestamp = 5;
  float confidence = 6;
  bool stable = 7;
}

• double rmssd -- root mean square of successive differences between normal heartbeats
• double mean_nn -- mean normal-to-normal (NN) interval length
• double sdnn -- Standard Deviation of normal-to-normal (NN) Intervals
• double baevsky -- Baevsky's Stress Index: a measure of autonomic balance derived from the NN interval distribution, computed as AMo / (2 * Mo * MxDMn). Reported without a unit, matching the HRV model card.
• int64 timestamp -- Absolute timestamp at which the HRV measurement was taken, in microseconds, since Linux epoch
• float confidence -- Confidence score for the HRV measurement, expressed as a percentage in the range [0.0, 100.0]
• bool stable -- Whether the HRV measurement is considered stable/reliable

Container for strict/exact values that require high precision. Used when measurements need to be treated with special precision requirements.

message Strict {
  float value = 1;
}

• float value -- The strict value requiring high precision

Comprehensive pulse-related measurements and derived metrics. Contains heart rate, pulse trace, and respiratory coupling information.

message Pulse {
  repeated MeasurementWithConfidence rate = 1;
  repeated Measurement trace = 2;
  repeated Measurement pulse_respiration_quotient = 3;
  Strict strict = 4;
}

• repeated MeasurementWithConfidence rate -- Heart rate measurements with confidence scores
• repeated Measurement trace -- Raw pulse trace measurements
• repeated Measurement pulse_respiration_quotient -- Pulse-respiration quotient measurements indicating cardio-respiratory coupling
• Strict strict -- Strict/high-precision pulse measurements over a fixed time interval. Populated when strict mode analysis is enabled.

Comprehensive breathing/respiratory measurements and derived metrics. Contains respiratory rate, traces, and various breathing pattern indicators.

message Breathing {
  repeated MeasurementWithConfidence rate = 1;
  repeated Measurement upper_trace = 2;
  repeated Measurement lower_trace = 3;
  repeated Measurement amplitude = 4;
  repeated DetectionStatus apnea = 5;
  repeated Measurement respiratory_line_length = 6;
  repeated Measurement baseline = 7;
  repeated Measurement inhale_exhale_ratio = 8;
  Strict strict = 9;
}

• repeated MeasurementWithConfidence rate -- Respiratory rate measurements with confidence scores. stable marks confidence at or above the minimum accepted accuracy standard for breathing rate (+/-1 br/min), which corresponds to confidence >= 45; see the breathing model card for the confidence-to-error mapping.
• repeated Measurement upper_trace -- Chest breathing movement trace measurements. The trace has no confidence of its own; stable is inherited from the breathing rate verdict (>= 45); see the breathing model card.
• repeated Measurement lower_trace -- Abdominal breathing movement trace measurements. The trace has no confidence of its own; stable is inherited from the breathing rate verdict (>= 45); see the breathing model card.
• repeated Measurement amplitude -- Breathing amplitude measurements
• repeated DetectionStatus apnea -- Apnea (breathing cessation) detection status
• repeated Measurement respiratory_line_length -- Respiratory line length measurements for breathing pattern analysis
• repeated Measurement baseline -- Baseline breathing measurements
• repeated Measurement inhale_exhale_ratio -- Inhale to exhale duration ratio measurements
• Strict strict -- Strict/high-precision breathing measurements over a fixed time interval. Populated when strict mode analysis is enabled.

Facial landmark coordinates with temporal and stability information. Used for face tracking and facial feature analysis.

message Landmarks {
  repeated Point2dFloat value = 1;
  bool stable = 2;
  bool reset = 3;
  int64 timestamp = 4;
}

• repeated Point2dFloat value -- Array of 2D coordinate points representing facial landmarks
• bool stable -- Whether the landmark detection is considered stable/reliable
• bool reset -- Indicates whether the landmark set was reset (cannot be directly associated with previous set)
• int64 timestamp -- Absolute timestamp at which the landmarks were detected, in microseconds, since Linux epoch

Comprehensive facial analysis measurements and detections. Contains blinking, talking detection, landmarks, and expressions.

message Face {
  repeated DetectionStatus blinking = 1;
  repeated DetectionStatus talking = 2;
  repeated Landmarks landmarks = 3;
  repeated Expression expression = 4;
}

• repeated DetectionStatus blinking -- Blinking detection status over time
• repeated DetectionStatus talking -- Talking/speech detection status over time
• repeated Landmarks landmarks -- Facial landmark coordinates over time
• repeated Expression expression -- Detected expressions over time

Electrodermal Activity (EDA) measurements. Tracks skin conductance changes related to autonomic nervous system activity.

message Eda {
  repeated Measurement trace = 1;
}

• repeated Measurement trace -- EDA trace measurements over time

message Cardio {
  repeated MeasurementWithConfidence pulse_rate = 1;
  repeated MeasurementWithConfidence arterial_pressure_trace = 2;
  repeated Hrv hrv = 3;
}

• repeated MeasurementWithConfidence pulse_rate -- Heart rate measurements with confidence scores. stable marks confidence at or above the minimum accepted accuracy standard for heart rate (+/-3 bpm), which corresponds to confidence >= 40; see the arterial pressure model card for the confidence-to-error mapping. Heart rate is derived from the same signal.
• repeated MeasurementWithConfidence arterial_pressure_trace -- Arterial pressure trace (uncalibrated, unitless) measurements with confidence scores. stable uses the same threshold as heart rate -- the minimum accepted accuracy standard (+/-3 bpm), confidence >= 40 -- since both derive from the arterial pressure signal; see the arterial pressure model card.
• repeated Hrv hrv -- Heart rate variability measurements with confidence scores. stable marks confidence at or above the minimum accepted accuracy standard for HRV (+/-5 ms), which corresponds to confidence >= 50; see the HRV model card for the confidence-to-error mapping.

Comprehensive physiological metrics container. Contains all available physiological measurements and analysis results.

message Metrics {
  Breathing breathing = 1;
  Eda eda = 3;
  Face face = 4;
  Cardio cardio = 5;
}

• Breathing breathing -- Breathing and respiratory analysis results
• Eda eda -- Electrodermal activity measurements. Note: processing needs to run for over 35 seconds to generate the first EDA result.
• Face face -- Facial analysis results
• Cardio cardio -- Cardiovascular measurements (pulse rate, arterial pressure, HRV)

Represents a 2D point with integer coordinates. Used for pixel-based coordinate systems and discrete positioning.

message Point2dInt32 {
  int32 x = 1;
  int32 y = 2;
}

• int32 x -- X coordinate as 32-bit signed integer
• int32 y -- Y coordinate as 32-bit signed integer

Represents a 2D point with floating-point coordinates. Used for precise positioning, normalized coordinates, and sub-pixel accuracy.

message Point2dFloat {
  float x = 1;
  float y = 2;
}

• float x -- X coordinate as floating-point value
• float y -- Y coordinate as floating-point value

Represents a 3D point with floating-point coordinates. Used for spatial positioning, depth information, and 3D landmark representation.

message Point3dFloat {
  float x = 1;
  float y = 2;
  float z = 3;
}

• float x -- X coordinate as floating-point value
• float y -- Y coordinate as floating-point value
• float z -- Z coordinate (depth) as floating-point value